My AI Agent Journey: From Confusion to Creation

Hey there, agent-in-training! Emma here, back from another late-night deep explore the fascinating, sometimes frustrating, world of AI agents. If you’re anything like me, you’ve probably spent the last few months hearing whispers (or full-blown shouts) about AI agents and thought, “Okay, this sounds cool, but also… what is it, really, and how do I actually get one to do something for me?”

Well, you’re in the right place. Today, we’re not just going to talk about what AI agents *are* in theory. We’re going to roll up our sleeves and tackle a question that’s been bugging a lot of you (and me, until recently): How do I build a simple AI agent that can actually browse the internet for me and answer specific questions?

Forget the hype for a minute. We’re going for practical. We’re going for “I can actually make this work on my laptop.” Because, let’s be honest, seeing is believing, right?

My Own Browser Bot Blunders (and Breakthroughs)

When I first started tinkering with the idea of an agent that could use the internet, I had grand visions. I wanted it to research complex topics, compare prices across a dozen sites, and basically be my personal digital assistant. What I got, initially, was a lot of errors, a few broken web scrapers, and a deep sense of inadequacy. My first attempts were like trying to teach a toddler to operate a complex machinery – lots of enthusiasm, zero understanding of the underlying mechanics.

The biggest hurdle? Getting an AI model to interact with the internet in a meaningful, controlled way. It’s one thing to ask ChatGPT a question it already knows. It’s another entirely to ask it to go find new information, interpret it, and then act on it. That’s where the “agent” part really comes into play.

After a lot of trial and error, I realized the key wasn’t to build a super-complex, all-knowing entity right out of the gate. It was to start small, with a very specific task, and build up from there. And that’s exactly what we’re going to do today. We’re going to create a basic browser agent that can go to a specific website, extract some information, and report back.

What Even IS a Browser AI Agent?

Before we explore the code, let’s quickly clarify. An AI agent, at its core, is an AI program designed to perform a specific task autonomously. It observes its environment (in our case, the internet), makes decisions based on its goals, and then takes actions. A *browser* AI agent specifically uses web browsing capabilities as part of its toolkit.

Think of it like this: you tell your agent, “Find me the current weather in London.” Instead of just pulling from its internal knowledge, a browser agent would (ideally) open a web browser, navigate to a weather site, find the information, and then give it back to you. It’s the difference between asking a librarian a question they already know the answer to, and asking them to go find a book, read a specific chapter, and then summarize it for you.

For our simple agent, we’ll be using a few common Python libraries that make this whole process a lot less intimidating.

The Tools We’ll Need

You don’t need a supercomputer for this. Just your regular development setup. Here’s what we’ll be using:

• Python: Our programming language of choice. (I’m using 3.9, but 3.8+ should be fine.)
• requests: A fantastic library for making HTTP requests (i.e., fetching web pages).
• BeautifulSoup (bs4): This is our web parsing wizard. It helps us navigate and extract data from HTML.
• A Large Language Model (LLM) API: We’ll need access to an LLM like OpenAI’s GPT models or Anthropic’s Claude. For this tutorial, I’ll assume you have an OpenAI API key.

First things first, let’s get our environment ready. If you don’t have these installed, open your terminal or command prompt and run:

pip install requests beautifulsoup4 openai

And make sure you have your OpenAI API key handy. You’ll need to set it as an environment variable or pass it directly in your script. For simplicity, I’ll show it directly, but for production, environment variables are always better.

Our Mission: Find a Specific Article on a Blog

Let’s set a concrete, achievable goal. Our agent’s mission will be to:

1. Go to a specific blog (let’s use a dummy blog URL for now, or even agent101.net if you’re feeling adventurous!).
2. Find a blog post title that contains a specific keyword (e.g., “beginner”).
3. Extract the URL of that blog post.
4. Summarize the content of that blog post using an LLM.

This is a fantastic starting point because it combines web interaction with LLM understanding. It’s like asking your digital assistant to “find me the latest article about AI agents for beginners on agent101.net and tell me what it’s about.”

Step 1: Fetching the Web Page

The first step for any browser agent is to “see” the web page. We’ll use the requests library for this. Let’s make a simple script:

import requests

def fetch_webpage(url):
 try:
 response = requests.get(url)
 response.raise_for_status() # Raises an HTTPError for bad responses (4xx or 5xx)
 return response.text
 except requests.exceptions.RequestException as e:
 print(f"Error fetching URL {url}: {e}")
 return None

# Let's use a placeholder URL for now.
# Replace with a real blog URL that has articles if you want to test!
blog_url = "https://www.example-blog.com/articles" # Use a real URL here!

html_content = fetch_webpage(blog_url)

if html_content:
 print("Successfully fetched the webpage content.")
 # We'll process this content in the next step
else:
 print("Failed to fetch webpage.")

A quick note on URLs: For this to work, you’ll need a real blog URL. If you don’t have one handy, you can use a site like `http://quotes.toscrape.com/` for testing simple scraping, but for blog posts, find an actual blog. Even this very site, agent101.net, could work if you adapt the CSS selectors later!

Step 2: Parsing the HTML and Finding Article Links

Now that we have the raw HTML, it’s like having a giant book without an index. BeautifulSoup is our indexer. We need to tell it how to find the article titles and their corresponding links.

This is the trickiest part because every website’s HTML structure is different. You’ll need to “inspect” the element on the target website. Right-click on an article title on your chosen blog and select “Inspect” (or “Inspect Element”). Look for common patterns like <h2> tags for titles, or <a> tags for links, often nested within a <div> with a specific class name.

For demonstration, let’s assume a common blog structure where article titles are in <h2> tags, and their links are in an <a> tag directly inside or preceding them.

from bs4 import BeautifulSoup

def find_article_links(html_content, keyword):
 soup = BeautifulSoup(html_content, 'html.parser')
 article_data = []

 # This is a generic example. You'll need to adjust selectors based on the target website.
 # Common patterns: div with class 'article-item', h2 with class 'article-title', etc.
 # Let's assume articles are in 'div' elements with class 'post'
 # and titles are in 'h2' tags within those divs, with a link inside.
 
 # Example: Look for all 
 tags that might contain article titles
 # Then check if they are linked.
 
 # A more solid approach might be to find containers first:
 article_containers = soup.find_all('div', class_='article-card') # Adjust this class!
 if not article_containers:
 # Fallback or try another selector if the first one doesn't work
 article_containers = soup.find_all('article') # Another common tag
 
 for container in article_containers:
 title_tag = container.find('h2') # Or 'h3', 'a', etc.
 link_tag = container.find('a', href=True) # Find the first link within the container
 
 if title_tag and link_tag:
 title = title_tag.get_text(strip=True)
 href = link_tag['href']
 
 # Make sure the URL is absolute if it's relative
 if href.startswith('/'):
 full_url = requests.compat.urljoin(blog_url, href)
 else:
 full_url = href

 if keyword.lower() in title.lower():
 article_data.append({'title': title, 'url': full_url})
 
 return article_data

# ... (previous fetch_webpage code) ...

if html_content:
 search_keyword = "beginner" # What article are we looking for?
 found_articles = find_article_links(html_content, search_keyword)

 if found_articles:
 print(f"\nFound articles containing '{search_keyword}':")
 for article in found_articles:
 print(f"- Title: {article['title']}\n URL: {article['url']}")
 # For this example, let's just take the first one found
 target_article = found_articles[0]
 else:
 print(f"\nNo articles found containing '{search_keyword}'.")
 target_article = None
else:
 target_article = None

Crucial Customization: The lines like soup.find_all('div', class_='article-card') and container.find('h2') are placeholders. You *must* adapt these based on the actual HTML structure of the blog you’re targeting. This is where inspecting the webpage becomes essential. My best advice is to start broad (e.g., `soup.find_all(‘a’)` to get all links) and then narrow it down with classes or IDs.

Step 3: Extracting Content from the Target Article

Once we have the URL of our target article, we need to fetch its content and then extract the main body text. This is often easier than parsing an index page, as most article content lives within a main content <div> or <article> tag.

# ... (previous code) ...
import openai
import os

# Set your OpenAI API key
# Best practice: os.environ.get("OPENAI_API_KEY")
# For this example, direct assignment:
openai.api_key = "YOUR_OPENAI_API_KEY" # REPLACE THIS WITH YOUR ACTUAL KEY!

def extract_article_text(article_url):
 article_html = fetch_webpage(article_url)
 if not article_html:
 return None

 soup = BeautifulSoup(article_html, 'html.parser')

 # This is highly dependent on the website's structure.
 # Common patterns: find a div with class 'entry-content', 'article-body', 'main-content'
 # Or just the 
 tags within the main article tag.
 
 # Let's try to find common content areas.
 content_div = soup.find('div', class_='entry-content') # Common for WordPress
 if not content_div:
 content_div = soup.find('article') # Another good general tag
 if not content_div:
 # As a last resort, just get all paragraph tags
 paragraphs = soup.find_all('p')
 return "\n".join([p.get_text(strip=True) for p in paragraphs if p.get_text(strip=True)])

 # If we found a specific content div/article, extract all paragraph text from it
 paragraphs = content_div.find_all('p')
 article_text = "\n".join([p.get_text(strip=True) for p in paragraphs if p.get_text(strip=True)])
 return article_text

if target_article:
 print(f"\nFetching content for: {target_article['title']}")
 article_text_content = extract_article_text(target_article['url'])
 
 if article_text_content:
 # print(article_text_content[:500]) # Print first 500 chars to verify
 print("\nSuccessfully extracted article text. Moving to summarization.")
 else:
 print("Failed to extract article text.")
 article_text_content = "" # Ensure it's not None for the next step
else:
 article_text_content = ""

Again, the soup.find('div', class_='entry-content') line is your primary target for customization. Use your browser’s inspector!

Step 4: Summarizing with an LLM

Finally, the moment we bring in the AI brain! We’ll feed the extracted text to an LLM and ask it . This is where the “intelligence” of our agent really shines.

# ... (previous code) ...

def summarize_text_with_llm(text):
 if not text:
 return "No text provided for summarization."
 
 # Truncate text if it's too long for the model's context window
 # GPT-3.5-turbo has a 16k context window, but it's good practice to keep it reasonable.
 # For a beginner, aiming for 4000-8000 tokens is safe.
 # Roughly 1 token = 4 characters for English text.
 max_tokens_for_input = 12000 # Adjust based on your model's capacity and cost considerations
 
 if len(text) > max_tokens_for_input * 4: # Crude character estimate
 text = text[:max_tokens_for_input * 4]
 print(f"Warning: Text truncated to ~{max_tokens_for_input} tokens for LLM processing.")

 try:
 response = openai.chat.completions.create(
 model="gpt-3.5-turbo", # Or "gpt-4", "claude-3-opus-20240229", etc.
 messages=[
 {"role": "system", "content": "You are a helpful assistant that summarizes blog posts concisely."},
 {"role": "user", "content": f"Please summarize the following blog post for me:\n\n{text}"}
 ],
 temperature=0.7, # Controls randomness. Lower for more focused summaries.
 max_tokens=500 # Max tokens for the summary itself
 )
 return response.choices[0].message.content.strip()
 except openai.APIError as e:
 print(f"OpenAI API Error: {e}")
 return "Failed due to API error."
 except Exception as e:
 print(f"An unexpected error occurred during summarization: {e}")
 return "Failed due to an unexpected error."

if article_text_content:
 print("\nRequesting LLM summarization...")
 summary = summarize_text_with_llm(article_text_content)
 print("\n--- Article Summary ---")
 print(summary)
else:
 print("\nCannot summarize: No article content available.")

And there you have it! A basic but functional AI agent that can browse, extract, and summarize. This isn’t just theory anymore; it’s a tangible piece of code you can run.

Putting It All Together (Full Script)

Here’s the complete script for easy copy-pasting and testing. Remember to replace placeholder URLs and your OpenAI API key!

import requests
from bs4 import BeautifulSoup
import openai
import os

# --- Configuration ---
# Replace with your actual blog URL. Make sure it has articles!
TARGET_BLOG_URL = "https://www.example-blog.com/articles" 
SEARCH_KEYWORD = "beginner" # The keyword to find in article titles
OPENAI_API_KEY = "YOUR_OPENAI_API_KEY" # REPLACE THIS! Or use os.environ.get("OPENAI_API_KEY")

# Set the OpenAI API key
openai.api_key = OPENAI_API_KEY

# --- Helper Functions ---
def fetch_webpage(url):
 """Fetches the HTML content of a given URL."""
 try:
 response = requests.get(url, timeout=10) # Added a timeout
 response.raise_for_status() 
 return response.text
 except requests.exceptions.RequestException as e:
 print(f"Error fetching URL {url}: {e}")
 return None

def find_article_links(html_content, base_url, keyword):
 """Parses HTML to find article titles and URLs matching a keyword."""
 soup = BeautifulSoup(html_content, 'html.parser')
 article_data = []

 # --- CUSTOMIZATION POINT 1: Adjust these selectors for your target blog ---
 # Inspect the blog page to find the correct HTML tags and classes for article containers, titles, and links.
 article_containers = soup.find_all('div', class_='article-card') # Common for article cards/previews
 if not article_containers:
 article_containers = soup.find_all('article') # Another common tag for individual articles
 if not article_containers:
 print("Warning: Could not find common article container tags. Trying broader search.")
 # Fallback: just look for all links that might be article links
 all_links = soup.find_all('a', href=True)
 for link in all_links:
 title_text = link.get_text(strip=True)
 if keyword.lower() in title_text.lower():
 href = link['href']
 full_url = requests.compat.urljoin(base_url, href)
 article_data.append({'title': title_text, 'url': full_url})
 return article_data # Return early if only broad links were found

 for container in article_containers:
 # Look for the title and link within each container
 title_tag = container.find(['h2', 'h3', 'a']) # Titles often in h2/h3 or directly linked
 link_tag = container.find('a', href=True)
 
 if title_tag and link_tag:
 title = title_tag.get_text(strip=True)
 href = link_tag['href']
 
 # Construct full URL if it's relative
 full_url = requests.compat.urljoin(base_url, href)

 if keyword.lower() in title.lower():
 article_data.append({'title': title, 'url': full_url})
 
 return article_data

def extract_article_text(article_url):
 """Fetches an article page and extracts its main textual content."""
 article_html = fetch_webpage(article_url)
 if not article_html:
 return None

 soup = BeautifulSoup(article_html, 'html.parser')
 
 # --- CUSTOMIZATION POINT 2: Adjust these selectors for the main article content ---
 # Look for the main content area of the article (e.g., div with class 'entry-content', 'article-body')
 content_area = soup.find('div', class_='entry-content') 
 if not content_area:
 content_area = soup.find('article', class_='main-article-content') # Another common pattern
 if not content_area:
 content_area = soup.find('div', id='content') # Yet another common ID

 if content_area:
 paragraphs = content_area.find_all('p')
 return "\n".join([p.get_text(strip=True) for p in paragraphs if p.get_text(strip=True)])
 else:
 # Fallback: get all paragraph tags if specific content area not found
 print(f"Warning: Specific content area not found for {article_url}. Extracting all paragraphs.")
 paragraphs = soup.find_all('p')
 return "\n".join([p.get_text(strip=True) for p in paragraphs if p.get_text(strip=True)])

def summarize_text_with_llm(text):
 """Uses an LLM the provided text."""
 if not text or len(text.strip()) < 50: # Minimum text length to attempt summarization
 return "Not enough content for summarization."
 
 # Simple truncation to stay within typical context window limits
 # Consider using a proper token counter for more accuracy if needed
 max_chars_for_llm = 40000 # Roughly 10k tokens for GPT-3.5-turbo 16k context
 if len(text) > max_chars_for_llm:
 print(f"Warning: Text truncated from {len(text)} to {max_chars_for_llm} characters for LLM.")
 text = text[:max_chars_for_llm]

 try:
 response = openai.chat.completions.create(
 model="gpt-3.5-turbo", # You can use "gpt-4" if you have access and want better quality
 messages=[
 {"role": "system", "content": "You are a helpful assistant that summarizes blog posts concisely and clearly."},
 {"role": "user", "content": f"Please provide a concise summary of the following blog post, highlighting key takeaways:\n\n{text}"}
 ],
 temperature=0.6, # A bit less random for summaries
 max_tokens=600 # Max tokens for the generated summary
 )
 return response.choices[0].message.content.strip()
 except openai.APIError as e:
 print(f"OpenAI API Error: {e}")
 return f"Failed due to API error: {e}"
 except Exception as e:
 print(f"An unexpected error occurred during summarization: {e}")
 return f"Failed due to an unexpected error: {e}"

# --- Main Agent Logic ---
def run_browser_agent():
 print(f"Starting browser agent for {TARGET_BLOG_URL} to find articles about '{SEARCH_KEYWORD}'...")

 # Step 1: Fetch the main blog page
 blog_html = fetch_webpage(TARGET_BLOG_URL)
 if not blog_html:
 print("Agent failed to fetch the main blog page. Exiting.")
 return

 # Step 2: Find articles matching the keyword
 found_articles = find_article_links(blog_html, TARGET_BLOG_URL, SEARCH_KEYWORD)

 if not found_articles:
 print(f"No articles found containing '{SEARCH_KEYWORD}' on {TARGET_BLOG_URL}. Agent finished.")
 return

 print(f"\nFound {len(found_articles)} potential articles containing '{SEARCH_KEYWORD}':")
 for i, article in enumerate(found_articles):
 print(f"{i+1}. Title: {article['title']}\n URL: {article['url']}")
 
 # For this example, let's process the first found article
 target_article = found_articles[0]
 print(f"\nProcessing the first found article: '{target_article['title']}' at {target_article['url']}")

 # Step 3: Extract content from the target article
 article_content = extract_article_text(target_article['url'])

 if not article_content:
 print(f"Agent failed to extract content from {target_article['url']}. Cannot summarize.")
 return

 print(f"\nExtracted article content (first 200 chars): {article_content[:200]}...")

 # Step 4: Summarize the content using LLM
 print("\nRequesting LLM the article...")
 summary = summarize_text_with_llm(article_content)

 print("\n--- Agent's Final Report ---")
 print(f"Article Title: {target_article['title']}")
 print(f"Article URL: {target_article['url']}")
 print("\nSummary:")
 print(summary)
 print("\n--- Agent Task Completed ---")

if __name__ == "__main__":
 run_browser_agent()

Actionable Takeaways for Your Own Agent Journey

This little project is just the tip of the iceberg, but it shows you the core mechanics. Here’s what I learned and what you should keep in mind:

1. Start Small, Think Specific: Don’t try to build Skynet on day one. Pick a very narrow, achievable task for your agent. “Find a recipe for vegan lasagna” is better than “cook dinner for me.”
2. Web Scraping is the Wild West: Every website is different. You WILL spend time inspecting elements in your browser to get the right CSS selectors or XPath. This is the grunt work, but it’s essential. Websites change, so your agent might need occasional tweaks.
3. Error Handling is Your Friend: Things will go wrong. Websites will be down, your internet will drop, API calls will fail. Add try-except blocks to gracefully handle these issues.
4. LLMs are Smart, but Need Guidance: The quality of your LLM’s output depends heavily on your prompt. Be clear, concise, and tell it exactly what you expect. Also, be mindful of context window limits and costs.
5. This is a Foundation: We’ve built a single-turn agent. Real agents often involve multiple steps, conditional logic (e.g., “if I find this, then do that”), and memory. But you’ve got the building blocks now.

My hope is that seeing this simple agent come to life demystifies the whole “AI agent” concept a bit. It’s not magic; it’s just intelligent automation built on existing tools. Go forth, tinker, and build your own little digital helpers!

Happy coding,

Emma Walsh, agent101.net

Related Articles

• How AI Agents Master Multiple Languages smoothly
• I Made My AI Agent Useful (Heres How)
• I Built an AI Agent in 2026: My Honest Take

🕒 Last updated: March 26, 2026  ·  Originally published: March 16, 2026