In Part 1 and Part 2, we covered the fundamentals and intermediate techniques. Now we’ll explore curl’s automation capabilities through globbing, handling various authentication schemes, and learn techniques for bypassing common restrictions.
These skills separate manual testing from efficient, automated reconnaissance. Whether you’re enumerating resources, dealing with authentication, or working around filters, these techniques will accelerate your assessments.
Understanding Compression
One time I exported a curl command from Burp and used curl to make the request. Other than the response headers, the body appeared to be empty, even though data was returned in Burp. After scratching my head for a few minutes, I noticed that the Content-Encoding header was set to gzip. I applied the --compressed flag and voila – there was the full response!
Many web servers send compressed responses (such as gzip) to reduce bandwidth. Without --compressed, curl shows raw compressed binary data that’s unreadable. With it, curl automatically requests compression and decompresses the response:
curl --compressed "https://www.example.com/"Globbing: Automated Enumeration
Globbing is curl’s built-in pattern expansion feature. It lets you generate multiple URLs from a single command, making resource enumeration very efficient. All you need is a couple of brackets or braces and you’re good to glob.
Basic Numeric Ranges
Test sequential resources like user IDs, document numbers, or backup files.
Let’s say you’re testing if user IDs 1-100 exist:
curl -I "https://www.example.com/api/users/[1-100]"This command will check https://www.example.com/api/users/1, https://www.example.com/api/users/2, and so on.
Let’s say you want to look for backups from multiple years:
curl -I "https://www.example.com/backup-[2020-2025].zip"You’ll be making requests for https://www.example.com/backup-2020.zip, https://www.example.com/backup-2021.zip, etc. with this command.
You can also step through ranges by specifying the count after a colon. For example, imagine you’re spot checking every 10 items in a list of 0-100:
curl -I "https://www.example.com/api/items/[0-100:10]"This will check 0, 10, 20, 30… all the way up to 100.
Alphabetic Ranges
You can test single-character parameters:
curl -I "https://www.example.com/files/document-[a-z].pdf"…and multiple character positions:
curl -I "https://www.example.com/files/[a-z][a-z].pdf"Pattern Lists with Braces
The real power comes from combining multiple patterns. The following can be used to test multiple directories and file extensions:
curl -I "https://www.example.com/{admin,backup,dev}/{config,settings}.{php,txt,bak}"This generates:
https://www.example.com/admin/config.phphttps://www.example.com/admin/config.txthttps://www.example.com/admin/config.bakhttps://www.example.com/admin/settings.phphttps://www.example.com/admin/settings.txthttps://www.example.com/admin/settings.bakhttps://www.example.com/backup/config.php- … and so on (for a total of 18 URLs)
Practical Enumeration Examples
In the first two examples, I’m going to introduce and use the -f (or --fail )flag. This makes curl fail silently on HTTP errors (like 404), so it won’t create empty files for non-existent resources. Without -f , you’re going to create a bunch of files containing the HTTP 404 response as its contents.
Look for and grab exposed configuration files with the remote file name:
curl -sfO "https://www.example.com/{config,configuration,settings,app}.{php,json,yml,yaml,xml,bak,old,txt}"Do the same as above, but with backups:
curl -sfO "https://www.example.com/{backup,db,database,sql,dump,site}.{sql,zip,tar.gz,tar,bak,old}"These command will download the discovered file(s) only if they were found.
Test common admin paths:
curl -I "https://www.example.com/{admin,administrator,wp-admin,phpmyadmin,manager,console}/"Enumerate API versions:
curl -I "https://www.example.com/api/v[1-5]/users"Combining Globbing with Output Formatting
Remember -w from Part 2? Combine it with globbing for efficient scanning. Here you can check for user IDs and show status codes:
curl -s -o /dev/null \
-w "User %{url_effective} - HTTP %{http_code}\n" \
"https://www.example.com/api/users/[1-50]"Here’s what the output would look like:
User https://www.example.com/api/users/1 - HTTP 200
User https://www.example.com/api/users/2 - HTTP 404
User https://www.example.com/api/users/3 - HTTP 200
...When Globbing Beats Specialized Tools
Remember from Part 1: curl shines when you’re living off the land. If you’ve compromised a server and need to enumerate internal resources, globbing lets you do it with a tool that’s likely already there. While FFUF is better for heavy fuzzing with large wordlists, curl’s globbing is perfect for:
- Quick checks of predictable patterns
- Testing specific numeric ranges
- Situations where you can’t install additional tools
- Scripting simple enumeration tasks
Authentication Schemes
Curl supports various authentication methods natively.
Basic Authentication
The simplest HTTP authentication scheme that sends credentials as base64-encoded username:password pairs in the Authorization header:
curl -u username:password "https://www.example.com/api/secure"Digest Authentication
This is a challenge-response mechanism that doesn’t send passwords in plaintext. The server sends a nonce (random value), and the client hashes the credentials with this nonce before transmission. More secure than Basic auth against replay attacks, though it still relies on MD5 hashing:
curl --digest -u username:password "https://www.example.com/api/secure"NTLM Authentication
Common in Windows environments, Microsoft’s proprietary authentication protocol is commonly found in Windows environments and Active Directory networks. NTLM auth uses a three-way handshake and challenge-response mechanism. Note the double backslash to escape the domain separator:
curl --ntlm -u domain\\username:password "https://internal.company.local/api"Bearer Token Authentication
This is a modern token-based authentication where the client presents a token (often JWT or OAuth access token) that was previously obtained through an authentication flow. The server validates the token’s signature and claims without needing to store session state:
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." \
"https://www.example.com/api/data"API Key Authentication
API keys are used as simple authentication using a pre-shared secret key, either in a custom header or as a query parameter. This is common in public APIs for rate limiting and access control, though less secure than token-based auth since keys are long-lived and harder to rotate:
# In header
curl -H "X-API-Key: your-secret-key" "https://www.example.com/api/data"
# In query parameter
curl "https://www.example.com/api/data?api_key=your-secret-key"Testing for Authentication Bypass
Of course we’ll always want to test if authentication is properly enforced.
Test without credentials:
curl -I "https://www.example.com/api/secure"Try different HTTP methods:
curl -I -X POST "https://www.example.com/api/secure"Bypassing Restrictions and Filters
Web applications often implement restrictions. Here are some techniques to test and potentially bypass them.
Custom User-Agent Strings
Some applications block or behave differently based on the user agent. By default, curl sets the user agent to, well, curl and the version. You can mimic a different browser with -A:
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
"https://www.example.com/"Referer Header Manipulation
Test Referer header checking:
curl -H "Referer: https://www.example.com/admin/dashboard" \
"https://www.example.com/admin/users"X-Forwarded Headers
Test if the application trusts X-Forwarded-For by spoofing an internal IP:
curl -H "X-Forwarded-For: 127.0.0.1" \
-H "X-Real-IP: 127.0.0.1" \
"https://www.example.com/admin"Path Normalization Bypass
By default, curl will normalize a path before sending your request to the server. This means that you won’t be able to test for path traversal. You can use --path-as-is to prevent curl from normalizing paths, leaving the ../ intact:
curl --path-as-is "https://www.example.com/users/../../admin"
This tests path traversal where the application and web server handle paths differently.
HTTP Method Override
Don’t forget to test for HTTP method override support. For example, you can use the following to send a POST, but override to DELETE:
curl -X POST \
-H "X-HTTP-Method-Override: DELETE" \
"https://www.example.com/api/users/1"Quick Reference
New flags from Part 3:
--compressed: Request and decompress compressed responses{a,b,c}: Brace expansion for multiple values[1-100]: Numeric range expansion[a-z]: Alphabetic range expansion-f, --fail: Makes curl fail silently on HTTP errors. Use this when downloading existing remote files only.--digest: Use Digest authentication--ntlm: Use NTLM authentication--path-as-is: Don’t normalize URL paths
Conclusion
Wow, I can’t believe we’re already at the end of our three-part curl series. You now have the foundation for using curl effectively in penetration testing: from basic requests and headers, through cookies and timing attacks, to automation with globbing and authentication handling.
The key to mastery is practice. Start incorporating these techniques into your daily testing workflow. You’ll find that curl becomes one of your most-reached-for tools, especially when you need quick answers or you’re working in constrained environments.
Remember: the best tool is the one that’s always available and that you know how to use effectively. Curl checks both boxes.
Thanks for reading, and good luck on your future engagements!