Curl

From Dikapedia
Jump to: navigation, search


curl - transfers a URL. Or make HTTP requests (or HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELENET, LDAP or FILE).


https://www.thegeekstuff.com/2012/04/curl-examples/


  • Useful for testing web servers at the application level, testing an application's endpoint or connectivity, or checking if a service is healthy.
  • HTTP uses TCP so if curl works then TCP (and lower levels) must be working, too.

Can be used in many different ways:

  • Designed to work without user interaction, making it excellent for automation.
  • Offers proxy support, user authentication, FTP uploading, HTTP posting, SSL connections, cookies, file transfer resume, metalink and more.
  • Download, upload, and manage files.
  • Check email address
  • Update your status on some social media websites, etc.


My Top Curl Commands


-I, --head

(HTTP/FTP/FILE) Fetch the HTTP-header only! HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE file, curl displays the file size and last modification time only.
$ curl -I dikapedia.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 30 Jan 2020 00:39:40 GMT
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Location: https://dikapedia.com/
Content-Type: text/html; charset=iso-8859-1

$ curl -I google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Thu, 30 Jan 2020 00:39:33 GMT
Expires: Sat, 29 Feb 2020 00:39:33 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN


-v, --verbose

Makes the fetching more verbose/talkative. Mostly useful for debugging. A line starting with '>' means "header data" sent by curl, '<' means "header data" received by curl that is hidden in normal cases, and a line starting with '*' means additional info provided by curl.

-L, --location

(HTTP/HTTPS) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, --include or -I, --head, headers from all requested pages will be shown.

$ curl -Iv 

$ curl -Iv dikapedia.com
* About to connect() to dikapedia.com port 80 (#0)
*   Trying 34.195.120.102...
* Connected to dikapedia.com (34.195.120.102) port 80 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Thu, 30 Jan 2020 00:45:43 GMT
Date: Thu, 30 Jan 2020 00:45:43 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< Location: https://dikapedia.com/
Location: https://dikapedia.com/
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1

<
* Connection #0 to host dikapedia.com left intact


$ curl -IvL:

  • Notice the 200 HTTP response at the end.
$ curl -IvL dikapeida.com
* Could not resolve host: dikapeida.com; Unknown error
* Closing connection 0
curl: (6) Could not resolve host: dikapeida.com; Unknown error
[ec2-user@ip-172-31-0-84 ~]$ curl -IvL dikapedia.com
* About to connect() to dikapedia.com port 80 (#0)
*   Trying 34.195.120.102...
* Connected to dikapedia.com (34.195.120.102) port 80 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently 
HTTP/1.1 301 Moved Permanently
< Date: Thu, 30 Jan 2020 00:46:24 GMT
Date: Thu, 30 Jan 2020 00:46:24 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< Location: https://dikapedia.com/
Location: https://dikapedia.com/
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1 

<
* Connection #0 to host dikapedia.com left intact
* Issue another request to this URL: 'https://dikapedia.com/'
* Found bundle for host dikapedia.com: 0x1539f70
* About to connect() to dikapedia.com port 443 (#1)
*   Trying 34.195.120.102...
* Connected to dikapedia.com (34.195.120.102) port 443 (#1) 
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=dikapedia.com
*       start date: Jan 20 20:07:12 2020 GMT
*       expire date: Apr 19 20:07:12 2020 GMT
*       common name: dikapedia.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Date: Thu, 30 Jan 2020 00:46:24 GMT
Date: Thu, 30 Jan 2020 00:46:24 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< Location: https://dikapedia.com/wiki/
Location: https://dikapedia.com/wiki/
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1 

<
* Connection #1 to host dikapedia.com left intact 
* Issue another request to this URL: 'https://dikapedia.com/wiki/'
* Found bundle for host dikapedia.com: 0x1539f70
* Re-using existing connection! (#1) with host dikapedia.com
* Connected to dikapedia.com (34.195.120.102) port 443 (#1)
> HEAD /wiki/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Thu, 30 Jan 2020 00:46:24 GMT
Date: Thu, 30 Jan 2020 00:46:24 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< X-Powered-By: PHP/7.2.24
X-Powered-By: PHP/7.2.24
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Vary: Accept-Encoding,Cookie
Vary: Accept-Encoding,Cookie
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private, must-revalidate, max-age=0
Cache-Control: private, must-revalidate, max-age=0
< Last-Modified: Thu, 30 Jan 2020 00:46:25 GMT
Last-Modified: Thu, 30 Jan 2020 00:46:25 GMT
< Location: http://dikapedia.com/wiki/index.php/All_Linux_Everythang~
Location: http://dikapedia.com/wiki/index.php/All_Linux_Everythang~
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8 

<
* Connection #1 to host dikapedia.com left intact
* Issue another request to this URL: 'http://dikapedia.com/wiki/index.php/All_Linux_Everythang~'
* Found bundle for host dikapedia.com: 0x1539f70 
* Re-using existing connection! (#0) with host dikapedia.com
* Connected to dikapedia.com (34.195.120.102) port 80 (#0)
> HEAD /wiki/index.php/All_Linux_Everythang~ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Thu, 30 Jan 2020 00:46:25 GMT
Date: Thu, 30 Jan 2020 00:46:25 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< Location: https://dikapedia.com/wiki/index.php/All_Linux_Everythang~
Location: https://dikapedia.com/wiki/index.php/All_Linux_Everythang~
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1 

<
* Connection #0 to host dikapedia.com left intact
* Issue another request to this URL: 'https://dikapedia.com/wiki/index.php/All_Linux_Everythang~'
* Found bundle for host dikapedia.com: 0x1539f70
* Re-using existing connection! (#1) with host dikapedia.com
* Connected to dikapedia.com (34.195.120.102) port 443 (#1)
> HEAD /wiki/index.php/All_Linux_Everythang~ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dikapedia.com
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 30 Jan 2020 00:46:25 GMT
Date: Thu, 30 Jan 2020 00:46:25 GMT
< Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
Server: Apache/2.4.41 () OpenSSL/1.0.2k-fips PHP/7.2.24
< X-Powered-By: PHP/7.2.24
X-Powered-By: PHP/7.2.24
< X-Content-Type-Options: nosniff 
X-Content-Type-Options: nosniff
< Content-language: en
Content-language: en
< Vary: Accept-Encoding,Cookie
Vary: Accept-Encoding,Cookie
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8 

<
* Connection #1 to host dikapedia.com left intact


Good one from john: get TTFB of dikapedia.com: 

$ curl -o /dev/null -s -w "Download Speed: %{speed_download} \n TTFB: %{time_starttransfer} \n Transfer Time:%{time_total}\n Download Size:%{size_download}\n" dikapedia.com
 Download Speed: 435.000 
 TTFB: 0.528569 
 Transfer Time:0.528630
 Download Size:230

Other commands


Check URL

One of the most common and simplest uses of curl.
This command will display the content of the URL on your terminal.
curl https://domain.com/


Save the output of the URL to a file

The output of a curl command can be easily saved to a file by adding the -o option to the command like so:
curl -o [file] https://domain.com


Download files

Download files with cURL by adding the -O to the command (remember from work)
curl -O https://domain.com/file.zip    (downloads to the current directory)
To download file with a different name by adding the -o option:
curl -o [newfilename] https://domain.com/file.zip
Download multiple files simultaneously:
curl -O https://domain.com/file.zip -O https://domain.com/file2.zip
Download files securely via SSH using:
curl -u user sftp://server.domain.com/path/to/file


Access an FTP server

To access your FTP server with cURL:
curl ftp://ftp.domain.com --user username:password
Curl will connect to the FTP server and list all files and directories in the user’s home directory.
Download files via FTP:
curl ftp://ftp.domain.com/filename.extension --user username:password.
Upload a file onto the FTP server
curl -T filename.extension ftp://ftp.domain.com/ --user username:passwor
Retrieved from "https://dikapedia.com/wiki/index.php?title=Curl&oldid=2285"