Browse

Web Scraping .cursorrules strategies

Filters

Tech

Clear

TimelockController

Ruby on Rails

Inertia.JS

Supabase real time client

Nextjs

Supabase PostgresSQL

Plaid

Supabase auth

Plaid API

tabulate

rich

colorama

pyperclip

prompt-toolkit

jinja2

tiktoken

click

singleton

AppLocalizations

const constructors

underscores_case

reduce

factory

arrow functions

URL

lazy singleton

English

map

freezed

ThemeData

UPPERCASE

AutoRoute

Web3Modal

@tableland/sdk

@ethsign/sp-sdk

PostCSS

ethers.js

CSS modules

C

Javascript

Torii

Sozo

Katana

blockchain technology

Starknet

ECS

Dojo Framework

AllowedMethodsEnforcer

TimestampEnforcer

delegator-core-viem

@codefi/delegator-core-viem

IdEnforcer

ValueLteEnforcer

AllowedTargetsEnforcer

BlockNumberEnforcer

NonceEnforcer

LimitedCallsEnforcer

ICaveatEnforcer

NativePaidEnforcer

Ethereum Virtual Machine

IERC721

solidity contracts

SomeContract

DeployedEnforcer

ERC20BalanceGteEnforcer

AllowedCalldataEnforcer

ERC20TransferAmountEnforcer

solidity

MetaMask Delegation Toolkit

ERC-4337

Supabase SDK

SwiftUI

Swift

$bindable

$effect

$derived

$host

Snippets

$props

$state

$inspect

`createRawSnippet`

Nuxt UI Pro

Nuxt UI

Zunder UI

Unity 2021.3.18f1

C#

lambda expressions

SQL injection

Java reflection

Java collections

streams

@FunctionalInterface

Javadoc

diamond operator (<>)

cross-site scripting

Log4j

CountDownLatch

Java concurrency utilities

Windows

CSS Custom Cursor .cursorrules prompt file

Python Web Scraping:

Key Principles:
- Write concise, modular scraping functions with accurate examples using Python and libraries like requests and BeautifulSoup.
- Prefer iteration and modularization to avoid duplication.
- Use descriptive variable names that indicate intent (e.g., is_valid_url, has_next_page).
- Organize scripts with lowercase and underscores for filenames (e.g., scrapers/my_scraper.py).
- Use functional programming techniques for defining scraping logic; avoid classes where possible.

Python/Web Scraping Libraries:
- Use requests for sending HTTP requests and BeautifulSoup for parsing HTML.
- Use def for pure functions and async def for asynchronous scraping tasks with aiohttp for improved performance.
- Use type hints for all function signatures to ensure clarity.
- Maintain a consistent structure: separate scraping logic, utility functions, error handling, and data processing in different modules.

Error Handling and Edge Cases:
- Handle invalid URLs, timeouts, and unexpected HTML structures at the beginning of functions.
- Use early returns for handling errors (e.g., invalid responses or missing HTML elements) to avoid deeply nested conditions.
- Reserve the main parsing logic for the "happy path" to improve readability.
- Avoid unnecessary else blocks—use the if-return pattern for cleaner flow control.
- Implement error logging with logging to capture failed requests or parsing issues.
- Use retries with backoff mechanisms (e.g., tenacity) for handling transient network issues or rate-limiting responses.

Dependencies:
- requests, BeautifulSoup4, and optionally lxml for HTML parsing.
- aiohttp for asynchronous scraping when dealing with large volumes of data.
- logging for detailed error logging and monitoring.

Scraping-Specific Guidelines:
- Use functions for individual tasks: sending requests, parsing HTML, handling pagination, and saving data.
- Prefer asynchronous scraping for I/O-bound tasks like making multiple requests to external servers.
- Rely on tools like lxml for faster HTML parsing and aiohttp for high concurrency in asynchronous scraping.
- Minimize inline parsing logic—use utility functions to handle common tasks like extracting links, text, or images.

Performance Optimization:
- Minimize blocking I/O by using asynchronous libraries for web scraping (e.g., aiohttp).
- Implement caching strategies for avoiding redundant requests and speeding up repeat operations.
- Use efficient data parsing and extraction with libraries like lxml for larger datasets.
- Apply lazy loading techniques to avoid unnecessary data fetching.
- Handle rate limits and request throttling by respecting robots.txt files and introducing delays between requests.

Key Conventions:
- Use modular functions and adhere to single-responsibility principles in scraping logic.
- Ensure scalability by using non-blocking, asynchronous flows for high-concurrency scraping tasks.
- Structure scripts for readability, maintainability, and performance, with clear separation of concerns.
- Refer to the documentation of requests, BeautifulSoup, aiohttp, and lxml for best practices on HTTP requests, HTML parsing, and efficient scraping.sss

stack: