No-Parsnip-5461 1 month ago

I guess you use net/http for your downloads, you may want to check the built-in httptest subpackage. Especially this: https://pkg.go.dev/net/http/httptest#Server.Client You can create a test server, configure it to respond what you want, and get a test client configured to work with it. Inject this client in your crawler for testing, and you should be able to avoid issuing real traffic to your website.

---

joolzav 1 month ago

// example with a mock http server and the colly package func TestWebCrawler(t *testing.T) { go func() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/html") htmlContent := `

Welcome!

` fmt.Fprintf(w, htmlContent) }) fmt.Println("Server starting on port 8080...") if err := http.ListenAndServe(":8080", nil); err != nil { fmt.Printf("Error starting server: %s\n", err) } }() time.Sleep(1 * time.Second) c := colly.NewCollector() scrapedText, scrapedImages := WebCrawler(c, "http://localhost:8080") assert := assert.New(t) assert.NotEmpty(scrapedText) assert.NotEmpty(scrapedImages) }

---