How to Scrape Competitor Reviews for Lead Gen (TrustRadius)

Hey everyone, Hans here. Last week I posted a scale on how you can scrape your competitors clients from Trust Radius and you can even filter for unhappy clients. Today I'm going to show you exactly how you can build a scale like that and what type of workflow actually powers that in the back end. So that if you want you can rebuild this cell you yourself from scratch or you can use the inspiration for any other scraping or workflows that maybe you want to do. If you've been following me for a while, then you know that I'm very excited to show you. So let's just dive straight in. Now. I asked Kerzer to explain the workflow a little bit and I will run you through this later. But, but first I want to show you what it looks like if you will be doing it manually because that's fairly straightforward. The idea is if you're on a website like TrustRadius, and I've built the same workflow for G2 as well for Capterra, the idea is straightforward. What you do is you find a competitor or a company that's interesting to you. I just took one example which is Celonis, in this case and then when you look at the reviews, so here for example verified use user. That's not something that we can, that we can really use. But here they give this person's full name and the company that he works at, the title that he has. Now in some cases that person still works there, still has that job title, etc. In some cases, they may have left already but we can then say, okay, find me, you know, the current company that they work at and you can still use that because we know exactly where they worked at which company and which title. Then here they also give you the exact company and here again they give you the exact company. Now that's different for every platform. So for G2 for example, they give you less information. For the platforms they give you different information. I'm just going to talk about Trust Radius for now and then in a different video we can dive a bit deeper into other platforms as well. But the idea is straightforward that we grab a page like this, we scrape it and then we want to find these people. Right? We have a lot of information. Now there are different ways that you can go about it, you can waterfall it, etc. I found that the easiest way of doing it is using Google AI mode. So here I just dropped, you know, a screenshot of that person. So Alice there, Peter Thompson, I Said, okay, find me this person a LinkedIn profile. Just like that, the LinkedIn profile for this person is. And they gave me a link to the actual LinkedIn profile of that person. So Google AI mode is actually really, really awesome at doing these types of searches. And the even better news is that scraping dog and other tools out there, they are very cheap when it comes to then scraping Google AI mode. So you basically get an AI agent Google AI mode and you can just easily, easily extract the you can extract the data from it. Now let me see. So they have here there's Google AI Mode. So it's 10 API credits, but their credits are very, very cheap. So basically what you do is they ask for the query. So I just recreate this query right here, where once the page is scraped we ask it for find, you know, Alice there, Peter Thompson, you know, find me this person's LinkedIn profile. And we do it for everyone on here. So we scrape this page, we skip the ones that say verified user, but then we say Nathan Bell, find me this person's LinkedIn profile. Beatrice, find me her LinkedIn profile. aiming the same. Now that way you can just run all those calls, scrape the page, run all those calls, you get all those people. And then what I do, in bulk once I have all those people scraped. And okay, these are people that, and you can even filter for date, you can filter for anything that you want, right. But you could say okay, find me ones that you know, left a one star, two star or three star review or left a five star review but give me their actual review so I can mention it, that type of thing. And once you have those, once I have everyone, they go into a revenue base. There's revenue based AI. We get their emails, their phone numbers and we can do the actual outreach. That is how straightforward it is to just get, you know, people from trust rages that have reviewed a tool that maybe is a competitor or is complementary to what it is that you're doing. Get their LinkedIn profiles, get their emails and reach out saying saw you said XYZ by HubSpot that maybe you didn't like this. We actually solved that. And what you could even do, you could obviously look for keywords, you can do it directly in the trustradius pages and scraping them is very cheap. Or you could try and Google for it, say okay, find me all the reviews that mention you know, like the cons about data setup for example, because we, you know, we are competitor of Celona our data setup is our core value proposition, or our visualization future is really good. So you start looking for that and then you can actually find people that are unhappy about the thing that you know your competitor isn't great at. And that's something that you actually solve. So that is how you scrape that. You use Scraping Dog to get the actual pages, to get them in markdown format. In this case for Trustradius, pagination is pretty straightforward. So you just change the URL, pages 2, 3, 4 etc. And, and you can just build like a really straightforward script to scrape all those pages until you get to the end. Once you have all these pages you then ask Google AI mode 5 minute LinkedIn profile. You get a response like this, which obviously isn't in a structured format but then you use like either regex or you know, Google Flash like some super cheap model and say okay, extract the LinkedIn profile from this response from Google AI mode, great. And then you have that and that goes on to revenue base and that is the entire workflow. So just to recap, and I've built it in a way that I can plug in a URL so I can say HubSpot.com, it goes out on Google finds, you know, HubSpot on TrustRadius, then actually grabs you know, the product URL and then we scrape it via Scraping Dog. Then we run through all the pages, we parse all the review cards for Markdown. As I said before, we get the title, the rating so we can potentially filter on that if we haven't done any filtering, on, on Trust Radius and then resolve them using Scraping Dog AI mode that actually Google AI mode, but we use Scraping Dog to scrape that. And then you know I build a table with it and they go all into go all into revenue base. So that is how you, one of the ways that you can find clients from your competitors and you can then say okay, let's just run this every month for example so we can get new reviews or new unhappy reviews and we reach out to those people. For me this is something that I usually run for clients or for use cases where it's low hanging fruit to find people that are very much, you know, aware of the solutions that are out there. You know, kind of using a competitor so they know exactly what's out there. We don't need to convince them about what it is that we're solving. We need to just need to convince them that our product is potentially better for them. And as I said, you can just run that repeatedly, so every month or every three months. And that's just a very interesting data source to always have in there. Hopefully you found this helpful. I will try and record one for G2 and for other platforms that are a little bit more tricky in the future as well. But for now, happy building and see you next time. Bye. Bye.