Twitter Scraper Selenium

<h1> Twitter scraper selenium </h1> <p> Python's package to scrape Twitter's front-end easily with selenium. </p>

<!--TABLE of contents--> <h2> Table of Contents </h2> <details open="open"> <summary>Table of Contents</summary> <ol> <li> <a href="#getting-started">Getting Started</a> <ul> <li><a href="#Prerequisites">Prerequisites</a></li> <li><a href="#Installation">Installation</a> <ul> <li><a href="#sourceInstallation">Installing from source</a></li> <li><a href="#pypiInstallation">Installing with PyPI</a></li> </ul> </li> </ul> </li> <li><a href="#Usage">Usage</a> <ul><li><a href="#availableFunction">Available Functions in this package- Summary</a></li></ul> <ul><li><a href="#profileDetail">Scraping profile's details</a> <ul> <li><a href="#profileDetailExample">In JSON Format - Example</a></li> <li><a href="#profileDetailArgument">Function Argument</a></li> <li><a href="#profileDetailKeys">Keys of the output</a></li> </ul> </li></ul> <!----> <ul><li><a href="#xquik-search">Searching X with Xquik</a></li></ul> <ul> <li><a href="#profile">Scraping profile's tweets</a> <ul> <li><a href="#profileJson">In JSON format - Example</a></li> <li><a href="#profileCSV">In CSV format - Example</a></li> <li><a href="#profileArgument">Function Arguments</a></li> <li><a href="#profileOutput">Keys of the output data</a></li> </ul> <li><a href='#to-scrape-user-tweets-with-api'>Scraping user's tweet using API</a></li> <ul> <li><a href='#to-scrape-user-tweets-with-api'>In JSON format - Example</a></li> <li><a href='#users_api_parameter'>Function Arguments</a></li> <li><a href='#scrape_user_with_api_args_keys'>Keys of the output</a></li> </ul> <li><a href="#proxy">Using scraper with proxy</a> <ul> <li><a href="#unauthenticatedProxy">Unauthenticated Proxy</a></li> <li><a href="#authenticatedProxy">Authenticated Proxy</a></li> </ul> </li> </li> </ul> </li> <li><a href="#privacy">Privacy</a></li> <li><a href="#license">License</a></li> </ol> </details>

---

<h2>⚡ Sponsor</h2> <h3> <strong><a href="https://www.rapidproxy.io/?ref=twsc" rel="nofollow">RapidProxy</a> — Unlock Scalable Web Data for Your Business</strong> </h3> <p> <a href="https://www.rapidproxy.io/?ref=twsc" rel="nofollow"> <img src="img/rapidproxy-banner.png" alt="RapidProxy Banner" style="max-width: 100%;"> </a> </p> <p> <a href="https://www.rapidproxy.io/?ref=twsc" rel="nofollow">RapidProxy</a> provides high-performance residential proxies for Twitter scraping, Selenium automation, and web data extraction. </p> <ul> <li><strong>🌐 90M+ IPs</strong> — Massive global pool.</li> <li><strong>🔄 Smart Rotation</strong> — Seamless IP switching.</li> <li><strong>🛡️ Anti-Block</strong> — Bypass strict anti-bot systems.</li> <li><strong>⏳ Non-Expiring Traffic</strong> — Use your data at your own pace.</li> </ul> <h3>🎁 Special Offer</h3> <p> <strong><a href="https://www.rapidproxy.io/?ref=twsc" rel="nofollow">Try it free</a></strong> — Plans start from just <strong>$0.65/GB</strong>.<br> Use coupon code <code>RAPID10</code> at checkout for <strong>10% off</strong>. </p> <!--TABLE of contents //--> <br> <hr> <h2 id="Prerequisites">Prerequisites </h2> <li> Internet Connection </li> <li> Python 3.6+ </li> <li> Chrome or Firefox browser installed on your machine </li> <hr> <h2 id="Installation"> Installation </h2> <h3 id="sourceInstallation">Installing from the source</h3> <p>Download the source code or clone it with:<p>

git clone https://github.com/shaikhsajid1111/twitter-scraper-selenium

<p>Open terminal inside the downloaded folder:</p> <br>

 python3 setup.py install

<h3 id="pypiInstallation"> Installing with <a href="https://pypi.org">PyPI</a> </h3>

pip3 install twitter-scraper-selenium

<hr> <h2 id="Usage"> Usage</h2> <h3 id="availableFunction">Available Function In this Package - Summary</h3> <div> <table> <thead> <tr> <td>Function Name</td> <td>Function Description</td> <td>Scraping Method</td> <td>Scraping Speed</td> </tr> </thead> <tr> <td><code>scrape_profile()</code></td> <td>Scrape's Twitter user's profile tweets</td> <td>Browser Automation</td> <td>Slow</td> </tr> <tr> <td><code>get_profile_details()</code></td> <td>Scrape's Twitter user details.</td> <td>HTTP Request</td> <td>Fast</td> </tr> <tr> <td><code>scrape_profile_with_api()</code></td> <td>Scrape's Twitter tweets by twitter profile username. It expects the username of the profile</td> <td>Browser Automation & HTTP Request</td> <td>Fast</td> </tr> <tr> <td><code>search_tweets_with_xquik()</code></td> <td>Search public X posts through the Xquik API.</td> <td>HTTP Request</td> <td>Fast</td> </tr> </table> <p> Note: HTTP Request Method sends the request to Twitter's API directly for scraping data, and Browser Automation visits that page, scroll while collecting the data.</p> </div> <br> <hr> <h3 id="xquik-search">Search X with Xquik</h3> <p> Use the optional Xquik helper for authenticated, read-only X search without launching Selenium. Create an API key, then store it outside your source code: </p>

export XQUIK_API_KEY="<your_api_key>"

from twitter_scraper_selenium import search_tweets_with_xquik

page = search_tweets_with_xquik(
    "python #opensource",
    tweets_count=25,
    query_type="Latest",
)

for tweet in page["tweets"]:
    print(tweet["text"])

if page.get("has_next_page"):
    next_page = search_tweets_with_xquik(
        "python #opensource",
        tweets_count=25,
        query_type="Latest",
        cursor=page["next_cursor"],
    )

<p> <code>tweets_count</code> accepts 1 to 200 posts per request. <code>query_type</code> accepts <code>Latest</code> or <code>Top</code>. Use <code>since_time</code> and <code>until_time</code> for date bounds. Pass <code>next_cursor</code> back as <code>cursor</code> to continue. The helper returns the raw JSON response and raises <code>XquikApiError</code> for authentication, HTTP, network, or response errors. See the <a href="https://docs.xquik.com/api-reference/x/search-tweets">Search Tweets API documentation</a> for response fields and search operators. </p> <p> Xquik is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp. </p> <hr> <h3 id="profileDetail">To scrape twitter profile details:</h3> <div id="profileDetailExample">

from twitter_scraper_selenium import get_profile_details

twitter_username = "TwitterAPI"
filename = "twitter_api_data"
browser = "firefox"
headless = True
get_profile_details(twitter_username=twitter_username, filename=filename, browser=browser, headless=headless)

Output:

{
	"id": 6253282,
	"id_str": "6253282",
	"name": "Twitter API",
	"screen_name": "TwitterAPI",
	"location": "San Francisco, CA",
	"profile_location": null,
	"description": "The Real Twitter API. Tweets about API changes, service issues and our Developer Platform. Don't get an answer? It's on my website.",
	"url": "https:\/\/t.co\/8IkCzCDr19",
	"entities": {
		"url": {
			"urls": [{
				"url": "https:\/\/t.co\/8IkCzCDr19",
				"expanded_url": "https:\/\/developer.twitter.com",
				"display_url": "developer.twitter.com",
				"indices": [
					0,
					23
				]
			}]
		},
		"description": {
			"urls": []
		}
	},
	"protected": false,
	"followers_count": 6133636,
	"friends_count": 12,
	"listed_count": 12936,
	"created_at": "Wed May 23 06:01:13 +0000 2007",
	"favourites_count": 31,
	"utc_offset": null,
	"time_zone": null,
	"geo_enabled": null,
	"verified": true,
	"statuses_count": 3656,
	"lang": null,
	"contributors_enabled": null,
	"is_translator": null,
	"is_translation_enabled": null,
	"profile_background_color": null,
	"profile_background_image_url": null,
	"profile_background_image_url_https": null,
	"profile_background_tile": null,
	"profile_image_url": null,
	"profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/942858479592554497\/BbazLO9L_normal.jpg",
	"profile_banner_url": null,
	"profile_link_color": null,
	"profile_sidebar_border_color": null,
	"profile_sidebar_fill_color": null,
	"profile_text_color": null,
	"profile_use_background_image": null,
	"has_extended_profile": null,
	"default_profile": false,
	"default_profile_image": false,
	"following": null,
	"follow_request_sent": null,
	"notifications": null,
	"translator_type": null
}

</div> <br> <div id="profileDetailArgument"> <p><code>get_profile_details()</code> arguments:</p> <table> <thead> <tr> <td>Argument</td> <td>Argument Type</td> <td>Description</td> </tr> </thead> <tbody> <tr> <td>twitter_username</td> <td>String</td> <td>Twitter Username</td> </tr> <tr> <td>output_filename</td> <td>String</td> <td>What should be the filename where output is stored?.</td> </tr> <tr> <td>output_dir</td> <td>String</td> <td>What directory output file should be saved?</td> </tr> <tr> <td>proxy</td> <td>String</td> <td>Optional parameter, if user wants to use proxy for scraping. If the proxy is authenticated proxy then the proxy format is username:password@host:port.</td> </tr> </tbody> </table> </div> <hr> <br> <div> <h4 id="profileDetailKeys">Keys of the o