How to SourceExtract the entire html code using CRAWL4AI? i tryed but it ouput only only visible webpage text

vittorianicolosi1994

Regular Member
Joined
Jan 17, 2024
Messages
279
Reaction score
81
code:[
!pip install -U crawl4ai
!pip install nest_asyncio

# Check crawl4ai version
import crawl4ai
print(crawl4ai.__version__.__version__)

##### Setup Crawl4ai
The following command installs Playride and its dependencies and updates a few configurations for Crawl4ai. After that, you can run the doctor command to ensure everything works as it should.


%%capture
!crawl4ai-setup

!crawl4ai-doctor

# If you face with an error try it manually
# !playwright install --with-deps chrome # Recommended for Colab/Linux

I suggest you first try the code below to ensure that Playwright is installed and works properly.

import asyncio
import nest_asyncio
nest_asyncio.apply()




from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def fetch_html():
url = "https://www.duelingbook.com/deck?id=12928546" # Replace with your target URL
config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url=url, config=config)
print("Raw HTML Content:")
print(result.html) # Directly access .html attribute

# Run the crawler
asyncio.run(fetch_html())
]
after i compare with entire html source and it isnt the same .crawl4ai ouptput only visible text
 
Crawl4ai is made to retrieve the visible parts of the webpages, not the source code,
Just use the typical requests library for the source code, it should do the work,

Here's an example:
Code:
import requests

url = "https://www.duelingbook.com/deck?id=12928546"
response = requests.get(url)

if response.status_code == 200:
    raw_html = response.text
    print("Raw HTML Content:")
    print(raw_html)
else:
    print(f"Failed to fetch the page. Status code: {response.status_code}")
 
Back
Top