how are searchbots reading my profile?

[PMc]

When I am not logged in, and I click on my name, I get this message:
You must be logged-in to do that

When I am logged in, and I click on my name, I can access a "profile page".

On that profile page I can post a message with a link.

When I do that, the next minute a couple of bots come for harvesting the link, e.g.:

***.***.**.**|18050 **********************************************************************
**.***.**.**|1966
***.***.**.***|65082 ***************************************************

One of these should be the forum itself fetching that OpenGraph stuff (which doesn't seem to fully work) - but none of them three look much like that.
What is going on?

 

[SirDice]

I checked the IPs against our user database. They belong to regular users, some of which you've interacted with. I'm not going to tell you which IP address belongs to which user though, for obvious reasons.

This is probably caused by the "unfurling" of the URLs.

 

[PMc]

I checked the IPs against our user database. They belong to regular users

Ah, indeed? Well, then that is fine - even if they have their own bots running, it is none of my business.

I'm not going to tell you which IP address belongs to which user though, for obvious reasons.

Obviousely, and I am removing the IP adresses also, they shouldn't be visible.

This is probably caused by the "unfurling" of the URLs.

Yes, that is it! "unfurling" means spicing these URLs up with an icon, a headline, an abstract text, a site info etc. (if the respective server provides such info).
Thats what I am testing, because basically I do not want it to look like this one here.
And the forum gets the abstract text somehow and display it, leaves space for an icon, sits there waiting for some two minutes, and then does remove the space for the icon again.
So I looked onto the webserver for whatever failing requests, and instead found these guys repeatedly showing that the icon can indeed be successfully downloaded.

What I didn't get rightwaway is: the forum somehow fetches the abstract text to be inserted into the page. But it doesn't fetch and cache the icon, it just inserts the image link as received, so the client's browser will finally fetch the icon, from wherever they are located.
So in that scenario I cannot see my own icons from all my servers in all my browsers, because of some hairpinning issues.

Thanks for the help!

 

[SirDice]

Thats what I am testing, because basically I do not want it to look like this one here.

Yeah, those are annoying to be honest. They're caused by that filter they've put up.

What I didn't get rightwaway is: the forum somehow fetches the abstract text to be inserted into the page. But it doesn't fetch and cache the icon, it just inserts the image link as received, so the client's browser will finally fetch the icon, from wherever they are located.

I'm a little fuzzy on how exactly this "unfurling" works, but I'm almost positive certain parts are done client-side. Maybe we can have the forums whitelisted, but I'm not sure if that would prevent those "Making sure you're not a bot" from happening.

One way is to explicitly create your URLs with bbcode instead of letting the forum code translate it. By default the forum code creates a [URL unfurl="true"]....[/URL] if you simply post a "plain" URL.

[url=https://example.com]This is my link to an example[/url].