- cross-posted to:
- technology@lemmy.world
- cross-posted to:
- technology@lemmy.world
Today, a prominent child safety organization, Thorn, in partnership with a leading cloud-based AI solutions provider, Hive, announced the release of an AI model designed to flag unknown CSAM at upload. It’s the earliest AI technology striving to expose unreported CSAM at scale.
You must log in or register to comment.
Man… That AI is going to be so fucked up when it gains sentience
Skynet’s real origin story. We might just deserve judgement day.
Oh we definitely do! Definitely for this, and definitely for many other things.
Jesus Christ. If someone ever got their hands on this model they could use it to generate new material. The grossest possible AI model to date
No. This is an inference model, not a generative model. You generally cannot train a model for both, unless you do it on purpose, and they certainly did not (especially since inference models are way easier to train than generative models).
A generative model uses the classifier as part of its training. If you generate a picture of pure random noise, then iteratively pick random noise that the classifier says “looks” more like csam, then you can effectively generate images that the classifier says it’s 100% certain is csam. Whether or not that looks anything like what a human would consider to be csam depends on other factors but it remains a possibility.
You are describing the way deepdream works, not the way modern Diffusion models work. It’s the difference between psychedelic dog faces and a highly adherent generative image of a German Sheppard.
I can’t imagine you’re going to get anything out of this model that actually looks like CSAM, unless there’s some sort of breakthrough in using these models for previously unrealized generative purposes.
I thought being able to do that was already a thing. This is designed to do the opposite.
I know, I know… bad actors and such.
me
no
rikeyThorn, the company backed by Ashton Kutcher and which tried to get its way to monitor all messages in the EU via Chat Control. No thanks.
Just remember folks. Kutcher is a slimeball too.
The guy went from a D list star and hanging out with the likes of Danny Masterson and going to Diddy’s infamous parties - to suddenly overnight courting the US government and being the face of ‘helping’ children everywhere.
Yeah right……
I’d be wary of calling him guilty by association. Maybe when he realized who he was really hanging out with he was so horrified and disgusted that he just had to get involved and do something to fight back?
Nah, it’s much easier to chastise people for not knowing what nobody knew
It’s awful coincidental that he seems to hang out with the ‘rapist’ crowd. Even going as far as writing a letter for Masterson as to how nice of a guy he is to try to get him a lenient sentence.
Even Hollywood has ostracized him and his wife - news sites recently reported they were looking to leave the country and let things cool off for a while.
I’m sure everyone is right though that keep posting here, that he is a swell guy who was just in the wrong place at the wrong time, multiple times. Several years worth of multiple times with wrong people. Just a coincidence.
The difference between us giving him a benefit of the doubt and claiming innocence and your take, is that you are labeling him a pedophile without proof. That’s a significant claim if false, and imo takes an assumption too far. Maybe he’s bad and it should be looked into, but saying he did something because he was on a show with and good friends with a guy that happened to be a rapist is wrong.
People can grow and change. Not saying he did or didn’t. Just saying that people aren’t a monolith. It’s plausible he just grew and his views changed / evolved.
That being said, it’s highly convenient where he’s positioned himself these days…
Regime whores are all about proximity
Has he ever called put the Catholic church or does he only care about pictures of abuse but not actual abuse?
Not a single peep about false positives.
I’m sure it won’t be abused though. And if anyone does complain, just get their electronics seized and checked, because they must be hiding something!
It could also, of course, make mistakes, but Kevin Guo, Hive’s CEO, told Ars that extensive testing was conducted to reduce false positives or negatives substantially. While he wouldn’t share stats, he said that platforms would not be interested in a tool where “99 out of a hundred things the tool is flagging aren’t correct.”
I take this to mean it is at least 1% accurate lol.
Reminds me of the A cup breasts porn ban in Australia a few years ago, because only pedos would watch that
This sort of rhetoric really bothers me. Especially when you consider that there are real adult women with disorders that make them appear prepubescent. Whether that’s appropriate for pornography is a different conversation, but the idea that anyone interested in them is a pedophile is really disgusting. That is a real, human, adult woman and some people say anyone who wants to live them is a monster. Just imagine someone telling you that anyone who wants to love you is a monster and that they’re actually protecting you.
There was a a porn studio that was prosecuted for creating CSAM. Brazil i belive. Prosecutors claimed that the petite, A-cup woman was clearly underaged. Their star witness was a doctor who testified that such underdeveloped breasts and hips clearly meant she was still going through puberty and couldn’t possible be 18 or older. The porn star showed up to testify that she was in fact over 18 when they shot the film and included all her identification including her birth certificate and passport. She also said something to the effect of women come in all shapes and sizes and a doctor should know better.
I can’t find an article. All I’m getting is GOP trump pedo nominees and brazil laws on porn.
I’m just glad they protected her
Pretty sure the adult star was lil Lupe. She was everywhere at the time because she did, indeed, look underage.
Awe man, I love all titties. Variety is the spice of life.
Not to mention the self image impact such things would have on women with smaller breasts, who (as I understand it) generally already struggle with poor self image due to breast size.
Clearly the state gives zero fucks about these women, or anyone else or even “the children”
Catholic Church is still around for a reason
Typically the state only cares about things they perceive as children.
Believe it or not, straight to jail.
If this is the price I must pay, I will pay it, sir! No man should be deprived of privately viewing a consenting adults perfectly formed small tit’s. They can take my liberty, they can take my livelihood, but they will never take away my boner for puffy nipples on a small chested half Japanese woman!
What is the charge? Biting a breast? A succulent Chinese breast?
Australia has a more general ban on selling or exhibiting hard porn, but is is legal to possess it. So it’s not just small boobs.
I am a bit confused how it is legal for them to have the training data here?
Like is there anything a corpo can’t do?
Like why can’t subway Jared and Catholic church “train the AI”
Only half way joking, what’s the catch here?
There are laws around it. Law enforcement doesn’t just delete any digital CSAM they seize.
Known CSAM is archived and analyzed rather than destroyed, and used to recognize additional instances of the same files in the wild. Wherever file scanning is possible.
Institutions and corporation can request licenses to access the database, or just the metadata that allows software to tell if a given file might be a copy of known CSAM.
This is the first time an attempt is being made at using the database to create software able to recognize CSAM that isn’t already known.
I’m personally quite sceptical of the merit. It may well be useful for scanning the public internet, but I’m guessing the plan is to push for it to be somehow implemented for private communication, no matter how badly that compromises the integrity of encryption.
So doesn’t that make the law enforcement having the biggest CP collection from everybody? This sounds kinda dangerous…
It does. Kinda.
The police are seldom allowed to be in possession of CSAM, except for in terms of grabbing the hardware which contains it in an arrest. The database used in modern detection tools is maintained by NCMEC which has special permission to do so.
And of course there are risks, but it’s just digital data. Unless you are creating more, you’re not actively harming anyone. And law enforcement absolutely needs that data to take some of the most obvious steps to prevent it being spread further.
Obviously, someone has access, but to get to the actual media files wouldn’t be simple. What typically happens, is that anyone wanting to detect CSAM, is given a hashed version of the database. They can then scan their systems for CSAM by hashing any media they are hosting, and seeing whether there are any matches.
Whenever possible, people aren’t handling the actual media. But for any detection to be possible to begin with, the database of the actual media does need to be maintained somewhere.
AI is a touchier subject, as you can’t train a model to recognize CSAM not already in the database using hashes, so in those cases you have to work with actual real media. This is only recently becoming a thing.
It also leaves open the possibility for false positives. An oft cited example is parents taking pictures of their own children for innocent reasons, or doctors and parents handling images for valid medical reasons. In a system that flagged such content, it would mean someone else would be seeing that “private” content because it was flagged.
I don’t think you even need the actual stuff to train a neural network to recognize it. For example, if I wanted to train a neural network to recognize pictures of lions, but I didn’t have any actual pictures of lions, I could use pictures of lion-shaped things, lion-colored things and locations where lions might appear. If a picture is hitting all three of those, it’s very likely to be a lion. Very likely is all a neural network can do, so it’s good enough for my purposes.
It’s the earliest AI technology striving to expose unreported CSAM at scale.
horde-safety has been out for a year now. Just saying… It’s not a trained AI model in this way, but it’s still using Neural Networks (i.e. “AI Technology”)
How did you figure out it had issues with broccoli?. Were you checking your vegetable gallery for CSAM?
haha, nah people reported some unexpected censors, and we investigated what part of their prompt might be causing it.
And will we get that technology to keep the Fediverse and free platforms safe? Probably not. All the predecessors have been kept away for sole use of the big players, despite populism always claiming we need to introduce total surveillance to keep the children safe…
IFTAS is already working with Thorn towards this goal. But you already have access to such technology through my toolset.
This one? I loosely followed your work… Maybe I should try it someday. See how it does on a regular VPS. Thanks for the link to the IFTAS. Seems they have curated some useful links… I’ll have a look at their articles. Hope they get somewhere with that. At this point, I don’t think there is any blocklist accessible to the average Fediverse admin?!
Edit: Thx, saw your other comment with the link to horde-safety.
Ye, a normal VPS would be too slow for production use, as a GPU is recommended. But you can plug in any home PC to do it without risks
Do you think this approach would be worth a try for the threaded Fediverse (aka Lemmy)? I mean your use-case is very different. We have some rudimentary image detection to flag other kinds of unwanted images in Piefed. I could experiment with something like https://github.com/monatis/clip.cpp. Have it go through the media cache and see if it can do something useful for us. But I don’t think it’d be worth all the effort unless the whole approach is somewhat accurate and runs in real time on average VPSes.
This approach was developed precicely for threaded fediverse. The initial use-case was protecting my own lemmy from CSAM! Check out fedi-safety and pictrs-safety
I was going to say… Sure would be nice to have this feature in all the open source AI image generator tools but you’re absolutely right 😩
Yeah, unless someone publishes even a set of hashes of known bad content for the general public… I kind of doubt the true intentions are preventing CSAM to the benefit of everyone.
If everyone has access to the model it becomes much easier to find obfuscation methods and validate them. It becomes an uphill battle. It’s unfortunate but it’s an inherent limitation of most safeguards.
You’re probably right. I’m not sure if it’s a good idea to walk close to the edge with things like this, though. Every update to the detection model could change things and get them in jail… So I certainly wouldn’t play a cat and mouse game with something that has several years of jailtime attached… But then I don’t really know the thought process of the average pedo. And AI image detection comes with problems anyways. In the article they say it detected 6 million pictures already. While keeping quiet about the rate of false positives. We know people have gotten in serious trouble for (false) claims. And I also wouldn’t want to be the Fediverse admin who has to go through thousands of flagged pictures and look at them and decide which is which. With consequences attached… Maybe a database of hashes would be the only option. That doesn’t detect new pictures, but at the same time it comes without flase positives and you can’t draw conclusions from hash values.
This seems like a potential actual good use of AI. Can’t have been much fun to train it though.
And is there any risk of people turning these kinds of models around and using them to generate images?
Available image generators are already capable of generating those images and they weren’t even trained on it. Once a neural network can detect/generate two separate concepts, it can detect/generate the overlap. It won’t be as fine-tuned obviously, but can still turn out scarily accurate.
If AI was reliable, maybe. MAYBE. But guess what? It turns out that “advanced autocomplete” does a shitty job of most things, and I bet false positives will be numerous.
This is not that kind of AI.
It’s possible to have a good AI system, but it takes millions of dollars and several thousand manhours to do, and most companies won’t put in the effort.
But, there should always be a human in the loop.
“detect new or previously unreported CSAM and child sexual exploitation behavior (CSE), generating a risk score to make human decisions easier and faster.”
False positives don’t matter if they stick to the stated intended purpose of making it easier to detect CSAM manually.
if they stick to the stated intended purpose
They never do.
The problem is that they won’t.
Yes, AI tools, in the hands of skilled people, can be very helpful.
But “AI” in capitalism doesn’t mean “more effective workers”, it means “fewer workers.” The issue isn’t technological so much as cultural. You fundamentally cannot convince an MBA not to try to automate away jobs.
(It’s not even a money thing; it’s about getting rid of all those pesky “workers rights” that workers like to bring with us)
Here’s the thing. This technology is unequivocally one of the things AI would be very useful for. It can potentially do a lot of good. Yes, MBAs could screw it up like they screw anything else up in society. That doesn’t mean we shouldn’t be happy that we’ve created this new tech.
And is there any risk of people turning these kinds of models around and using them to generate images?
There isn’t really much fundamental difference between an image detector and an image generator. The way image generators like stable diffusion work is essentially by generating a starting image that’s nothing but random static and telling the generator “find the cat that’s hidden in this noise.”
It’ll probably take a bit of work to rig this child porn detector up to generate images, but I could definitely imagine it happening. It’s going to make an already complicated philosophical debate even more complicated.
I think image generators in general work by iteratively changing random noise and checking it with a classifier, until the resulting image has a stronger and stronger finding of “cat” or “best quality” or “realistic”.
If this classifier provides fine grained descriptive attributes, that’s a nightmare. If it just detects yes or no, that’s probably fine.
Nobody would have been looking directly at the source data. The FBI or whoever provides the dataset to approved groups, but after that you just say “use all the images in this folder” and it goes. But I don’t even know if they actually provide real full-resolution images, or just perceptual hashes, or downsampled images.
And while it’s possible to use the dataset to generate new images assuming the training data had full-res images, like I said, I know they investigate the people making the request before allowing access. And access is probably supervised and audited.
This is a great development, albeit with a lot of soul crushing development behind it I assume. People who have to look at CSAM or whatever the acronym is have a miserable job, so I’m very supportive of trying to automate that away from people.
Yeah, I’m happy for AI to take this particular horrifying job from us. Chances are it will be overtuned (too strict), but if there’s a reasonable appeals process I could see it saving a lot of people the trauma of having to regularly view the worst humanity has to offer without major drawbacks.
I think all CSAM should be destroyed out of respect for the victims, not proliferated. I don’t care who is hanging onto this material or for what purpose.
How is this proliferating csam? Also, how do you expect them to find csam without having known images? It gives a really nice way to check based on hashes without having someone look at every picture on someone’s harddrive. With this AI it should greatly help determining new or unknown images while minimizing the number of actual people that have to see that stuff, and who get scarred from looking at such images. The only reason to be against this is if you are looking at CP and want it to be harder to find, or if you don’t understand how this technology is being used.
This ain’t about the victims… It never was, otherwise churches would NOT exist in current form.
This is about police and corpo state gaining power.
… robo chocolate?
At this point how does it differ w/ generating AI powered CP? morons
Uh, well this one tells you if an image looks like it or not. It doesn’t generate images
If it knows if an image looks like it it can generate something like it, one step further
Correct, this kind of software is trained on CP data. So such models can be easily used to generate CP instead of recognizing it, which makes them very dangerous indeed.
Same idea as the current models that are trained to recognized cars, these models can also be used to generate a car from noise as a starting poiint.
In pretttty sure you can’t just run it in reverse like that. There’s a whole different training and operation methodology you have to use to support generating images rather than simple text classification
There is a method of training where you use one system to make things and another to detect them. I forget the name of this approach, but it definitely is an approach.
It differs in basically being something completely different. This is a classification model, doesn’t have generative capabilities. Even if you were to get the model and it’s weights, and you tried to reverse engineer an “input” that it would classify as CP, it would most likely look like pure noise to you.
Moron
Generate porn, classificate output, result very young looking models.
Moron
So you need to have a model that generates CP to begin with. Flawless reasoning there.
Look, it’s clear you have no clue what you’re talking about. Stop demonstrating it, moron.
Alright, I found the name of what I was thinking of that sounds similar to what they’re suggesting: generative adversarial network (GAN).
The core idea of a GAN is based on the “indirect” training through the discriminator, another neural network that can tell how “realistic” the input seems, which itself is also being updated dynamically. This means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner.
Applying GAN won’t work. If used for filtering would result on results being skewed to a younger, but it won’t show 9 the body of a 9 year old unless the model could do that from the beginning.
If used to “tune” the original model, it will result on massive hallucination and aberrations that can result in false positives.
In both cases, decent results will be rare and time consuming. Anybody with the dedication to attempt this already has pictures and can build their own model.
Source: I’m a data scientist
At least it’s not “Source: I am a pedophile” lol
Not CP, but normal porn and select on CP traits, moron
https://en.m.wikipedia.org/wiki/False_positives_and_false_negatives
Not that I think you will understand. I’m posting this mostly for those moronic enough to read your comments and think “that seems reasonable”
Thanks
The model I use (I forget the name) popped out something pretty sus once. I wouldn’t describe it as CP, but it was definitely weird enough to really make me uncomfortable. It’s the only thing it ever made that I immediately deleted and removed from the recycling bin too lol.
The point I’m making is that this isn’t as far fetched as you believe.
Plus, you can merge models. Get a general purpose model that knows what children look like, a general purpose pornographic model, merge them, then start generating and selecting images based on Thorn’s classifier.
You can’t merge a generative model and a classification model. You can run then in series to get a bunch of false positives/hallucinations, but you can’t make it generate something from the other model.
When I said a “general purpose model that knows what children look like” I didn’t mean the classification model from the article. I meant a normal, general purpose image generation model. When I said “that knows what children look like” I mean part of its training set is on children, because it’s sort of trained a little on everything. When I said “pornographic model” I mean a model trained exclusively on NSFW content (and not including any CSAM, but that may be generous depending on how much care was out into the model’s creation).
