Golang web scraper NTLM authentication

Question

A Golang web scraper needs to extract information from a webpage that is NTLM-authenticated.

Having a valid username & password, how can the web scraper perform the NTLM 4-way handshake with the server in order to gain access to the protected webpage behind?

url, username, password := "http://www.some-website.com", "admin", "12345"

client := &http.Client{}
req, _ := http.NewRequest("GET", url, nil)
req.Header.Set("Authorization", "NTLM")
res, _ := client.Do(req)

Answer 1

You can use a package like Azure/go-ntlmssp to authenticate before you start scraping.

url, username, password := "http://www.some-website.com", "admin", "12345"

client := &http.Client{
    Transport: ntlmssp.Negotiator{
        RoundTripper:&http.Transport{},
    },
}

req, _ := http.NewRequest("GET", url, nil)
req.SetBasicAuth(username, password)

res, _ := client.Do(req)