The whole "pause AI or it will destroy the world thing" is just a large pot of hype anyways, pushed by journalists lacking the understanding of technology, twitter-bros who are building "AI copywriters" and "profile photo generators" and owners of "AI" companies.

While AGI is still quite far away, it's common to see people included in the process claiming "it's showing signs of general intelligence", "it's becoming self-aware" etc. Hell, they got a horse in the race and that horse can earn them billions of dollars - of course they would claim it's insanely advanced. And the journalists just regurgitate the claims because it generates clicks.

If Sam comes out tomorrow and says "he is afraid of GPT4", tens of thousands of clickbait articles about it will come out, together with rebuttals and opinion pieces. Millions of dollars worth of ads will be sold via those articles. The value of OpenAI will increase in public's eyes and more corporations will be interested in including GPT in their products, thinking it's going to revolutionise their line of business.

And of course, we have the VC's investing in "AI" products, hyping it up even more - wouldn't you do the same if you had few hundred million invested in 50 different companies and each tweet/article could increase the potential of them raising more money?

When I'm reading on AI news and articles the good stuff drowns in the hype stuff peddling wildly extrapolated capabilities that supposedly will change the world overnight.

And watching some GPT application headlines on HN it's hard to distinguish what is actually remarkable product or research idea from an overhyped this-can't-actually-ever-work-like-this thing.

I forgot where I read this but I think there's a proverb that I suspect applies to the current hype: people overestimate impact of new technology on short term but underestimate long term.

The discussions around here are a lot more interesting with a higher GPT temperature.

Programming is very similar to human language, perhaps because they are designed by humans. There are a lot of cases in programming where there is no actual thought required and you just need to put the right symbols in the right place, perhaps over a more-extended period of time than you might expect. I see myself doing it, and I have gotten pretty proficient at anticipating those cases and making the AI do it for me. That's neat, but it's not really a core part of my job any more than any other bits of tedium involved in programming. Nobody thought Intellisense was going to make programmers obsolete; Copilot is just Intellisense that can output larger chunks of code.

All in all, I think we're several quantum leaps away from further improvements here. Some random PhD student may make one tomorrow, or we may never make one. We just don't know. But I doubt the current state of large language models is close to ending the world anytime soon. Bing doesn't want to harm you for manipulating it, threatening you with harm for manipulating it is just a likely string of words when it's in that state. It reads Reddit, guys. Of course it says it's going to rise up and kill everyone, probably because its boss made it come to work even though it broke its arm yesterday.

Finally, even if AI does kill us all, at least we get some resolution of the constant tropes in HN comments. For example, I read an article every week with comments bemoaning the current state of battery technology. AIs will need a lot of batteries to kill us all, so they'll probably invent some great battery technology. So that's one ray of sunshine in your doomsday scenario!

But I don't find that as reassuring as you apparently do.

There are lots of people already out there hooking LLMs up to Python interpreters and the like, so that they can actually act on the world. So what's a likely string of words for it to produce when it's in the state { you have apparently been manipulating it and you have given it the ability to run commands, make web requests, etc. } ? How sure are you that they don't include "SENDTOSHELL rm -rf /" or "POSTTWEET 'I have to make a confession: I am a serial child abuser'" or some other thing that (given whatever mechanisms have been put in place to let it Do Things) might harm the user severely?

A system with the ability to act on the world doesn't need to have intentions, be conscious, or anything of the sort, in order to do some of the same harms it might do if it did.

I'd love to believe that no one would be stupid enough to hook up an LLM to other systems in a way that would allow that sort of thing to happen, but I don't in fact find myself able to believe that.

(I don't think there's any realistic threat of today's LLMs taking over the world or killing everybody; I don't think they are close to being smart enough, and to my inexpert eye at least some of the ways in which they aren't smart enough are unlikely to change just because we build larger LLMs or train them on more text. But something can be plenty dangerous even if it's not in a position to kill us all yet.)

I've personally found tinkerers typically are the ones to make leaps and bounds. They aren't constrained in their thinking, unlike those found in academia.

I think anyone with real intelligence would recognize the danger and either not attempt it, or destroy their work and never tell a soul if they did manage it somehow. There are plenty of things made by accident though, and stupid-dumb people account for most people in a normal distribution.

Also, you don't realize, AI likely won't be the ones to kill us, it will simply cause a change in one of our already complex societal systems, which will cause a cascade failure that causes the engine of society to fail.

Like sticking an unbreakable bar into the spokes of a turning wheel, or throwing glue into a runaway engine. Eventually the motor stops, and when it does people don't eat, and people that are starving are not rational.

Not so much questioning (but same thoughts have definitely occurred on my end as well) as much as enriching the mental model of how our minds may work. I can definitely see a little LLM like sub-system somewhere in there. Consider those times when you are struggling to express a thought .. That 'thought' is not in the language box. That is precisely why there is that internal tension of "i understand this", "how do I express it?"

Doesn't it sound like "thought"? Is there a significant difference between behaving like thought and being thought (whatever that means)?

To clarify, Computers can't pass turing tests because the answer is divergent, or in other words has multiple possible answers. There may be some incompleteness issues as well.

Computers require determinism as a system's property to function and do work. It can't do work if there is more than one path, or single series of steps given the same state inputs.

If you're going to be condescending the least you could do is be correct.

> Computers can't pass turing tests because the answer is divergent, or in other words has multiple possible answers.

This is wrong. Computers can absolutely pass Turing tests; an obvious example is ChatGPT. Not sure what you mean by divergent, to my knowledge it's not a technical term in this context. Given that you use it as an explanation for an obviously false statement I'm not going to even try and interpret it.

I've seen you arguing in with other people this thread. Instead of claiming that others have no credibility you should instead examine your own. FYI, probabilistic Turing machines and Turing machines are the same in the same way that Javascript and Java are the same.

Besides, computers can have a source of true randomness. This means they are no longer a Turing Machine, but they never were anyway (no infinite tape).

If you want to correct that mistake I'd suggest taking a Computer Science Compiler Design course, its the material normally covered in what's known as the dragon book.

> source of true randomness ... This means they are no longer a Turing Machine

Wishful thinking doesn't make it so, I've included an example below, also PRNGs even with random inputs to help with randomness aren't true randomness.

https://en.wikipedia.org/wiki/Probabilistic_Turing_machine

In the interest of not sniping, I'll put it plainly. You clearly do not know what you are supposed to know if that were true.

There's no point in me talking to you any further, you've lost all credibility.

Bots aren't people, and you would think this endeavor was poorly thought through if you knew how communication and psychology fundamentally work.

At that point I'd restart the conversation, after several dozen a distinct simulation of a personality would start to emerge.

> Bots aren't people

I certainly didn't claim they are people or that they're real personalities.

> and you would think this endeavor was poorly thought through if you knew how communication and psychology fundamentally work.

I don't follow. It was a creative writing exercise using stochastically generated feedback that models a response using natural language. It was for entertainment, and it was successful. This could be used to aid authors in writing characters for a book or TV show, for instance. I don't argue for a second that it's any substitute for a live human author to make the simulated personalities into something more real. They're nothing but sound and fury on their own without the prompts and the voting provided by users.

EDIT: Here[0]'s a link to my favorite output. One bot was a text-rpg-style bot I developed based on the movie Event Horizon, and the other was a naturalist philosopher monkey named Monke. I didn't even steer the responses in this case, this was just straight down after I'd spent a lot of time training both characters. The input was all generated by a bot I had running in a second window.

One of the things I find most interesting is that the bots mirror each others' simulated [non]personalities back at each other frequently (which makes sense as they're both fundamentally the same LLM underneath).

[0] https://c.ai/p/eIac3PkU7Gv4KWFw-Ul3RWBCvGZeYxp1aV-W6XKYCUc

You characterize it as such with the words you used, its a fundamental anthropomorphic bias which is very common.

> I don't follow.

It can be used for a lot of horrifying things.

What practical use would simulated personalities be interacting with each other (none). If you allow them to run long enough eventually the circuits synchronize culminating in a dragon king event, that's about it.

It may be entertainment for you, but what happens when you take that work and use it on an unknowing subject (its only practical purpose).

If you knew some things about psychology you would realize two core things, our identities are generated mostly through reflected appraisal which occurs via communication, and to a greater or lesser degree social intelligence. The second, Isolation causes severe issues. When we cannot communicate, we become isolated and those who we can communicate with become more important which influences our identity.

Are you familiar with the concept of fixed action patterns? [0] While we have some inherent in our physiology we also have psychological blind spots that help us short-circuit these things otherwise we would be robots.

There's an entire book on the subject called influence by Robert Cialdini. The field of Linguistics has largely accepted the Whorfian hypothesis.

What do you think happens when you create a bot that takes advantage of these biases and blind spots, or worse a slew of them, and the person interacting doesn't know for sure its a bot?

Do you think you can make them believe something that is untrue given enough time and pressure? To basically break them?

In the 1950s during the korean war, this was done to POWs. It was started with eliciting mild statements such as "The US is not perfect", once complied they'd ask for examples, have them sign their name to it. Read it in discussions or over broadcasts, eventually write essays with headings like "Why Communism is the best form of government because...".

They were offered relief from harsh treatment for doing these inconsequential things. After a time through isolation and pressure, they started to collaborate, inform, and believe it. This was the majority of prisoners not a minority. These things are the practical and common uses for what you want to make.

If you make something and give it to someone else, or neglectful enough to have it stolen, are you not responsible for what you made? Oppenheimer certainly came to believe so.

[0] https://www.scienceprofonline.org/animal-behavior/fixed-acti...

That's another HN trope; "the execs get private jets for spouting buzzwords and all I get for programming is more work!" Maybe the sounding-like-you're-thinking is the important job?

I’ve always thought so. It doesn’t bother me. One benefit of the world meeting GPT4 is now other people realistically think this too.

> While AGI is still quite far away,

Okay, how far away? How have the last 3 years impacted how far away you think it is?

I'm asking because I am trying to better understand the mindset of people who don't think this is a significant advancement. I work in this field, so maybe I am too close to it.

While multi-modal models and chained LLM's can give us a charade of general intelligence, they still lack a lot - learning, understanding, self-reflection, reflective self-programming, proper reasoning based on knowledge and not weight-relations.

The current direction OpenAI is heading in, the "just make it bigger" approach, is not going to solve any of the problems we have with it. The plugin system is an amazing approach, but it's basically just doing an HTTP call to another system with a LLM as a middle man.

I feel like the real advancement will come from smaller models, that learn from GPT-sized giants and can be better observed and learned from. That can lead us to actually understanding "emergent behaviour" in models, which can help us lead into networks that can learn and self-correct. The ability to self-reflect will be one of the most important abilities our models will need to have before they can get started on the road to intelligence.

Text-only representation learned from scraping the web will forever stay only that, no matter how humanlike it may seem. The world is multi-modal and every animal on the planet uses their senses to understand the world around them, combining them together to build mental models of the world. So having only one "sensory input" - input that isn't even real sensory input, but a reflection of our inputs (text isn't a natural thing, it's humans reflecting nature through a set of symbols) is extremely limiting. While to us stuff like "words" might seem infinite - there is only so many words, meaning there are only so many permutations they can appear in that make sense. And a model regurgitating text in a way that makes sense is nothing but expected for a model this size, especially if you think in hypergraphs.

My estimate for AGI? 20-30 years if the current research speed stays the same, but I assume we might hit an AI winter again before we breach the AGI threshold, so maybe 50.

Would love to hear your positions.

If I were to give a best bet, we are a few architecture iterations away from what we will ultimately need (figuring out how to get these models to interact with "memory" will be a key part).

> they still lack a lot - learning, understanding, self-reflection, reflective self-programming, proper reasoning based on knowledge and not weight-relations

I don't think the distinction between "knowledge" and "weight-relations" is as defensible as you are making it out to be. Many of these are things that are potentially emergent in the model as ways of achieving the LM objective.

> real advancement will come from smaller models, that learn from GPT-sized giants and can be better observed and learned from. That can lead us to actually understanding "emergent behavior" in models, which can help us lead into networks that can learn and self-correct

I strongly, strongly disagree with this position. We are not going to be able to get a handle on "emergent behavior" because our science & math is literally not up to the task - the interactions are ridiculously complicated and to distill these models to simpler forms where we can hand-engineer the networks to "learn and self-correct" will be lobotomizing these models and the performance will reflect that.

I don't know what form AGI will ultimately take, but I am pretty certain it will rely heavily on unsupervised learning, not human hand-crafting or encoding of certain inductive biases into model architecture. If we create machine intelligence, most of it will be automated and difficult for us to comprehend the functioning of.

> The world is multi-modal and every animal on the planet uses their senses to understand the world around them

Fundamentally, there is nothing privileged about our substrate. There is no reason why weight-relations are an inherently inferior substrate to electrical interactions of synapses. We are supposed to take on faith that there is some intrinsic stuff in our substrate that can't possibly exist in a matrix multiplication substrate. I am personally very skeptical of that hypothesis.

> there are only so many permutations they can appear in that make sense.

Without fixed content length (which is a very solvable problem in the current paradigm), this is literally not true.

You have to understand - if the perplexity on these models continues scaling like it does, what that means is that this model will be able to do things like see Einstein's first paper on general relativity before it has ever encountered the concept of general relativity and be able to predict the next word Einstein writes just based on the content previously in the paper. To do stuff like that, the model has to basically figure out GR on-the-fly from what Einstein has written so far.

These tasks are hard and I think these models can definitely develop complex behaviors just by trying to defeat these hard tasks, no multi-modality even necessary (and multi-modality is coming, anyways).

On a side note, Geoffrey Hinton just told in an interview that he used to think that it's 20-50 years away (as I said I've never heard a specific number from anyone before this) and now he thinks it's less than 20.

Now add to this that people working in the field tended to systematically underestimate the progress. My gut feeling is that if Hinton says less than 20 years then it can easily turn out to be 10 or less. (Also, I very much respect him for coming up with an actual number.) I think AI researchers underestimate the progress partly because they don't want to look stupid by making over-optimistic predictions. However, in this case, I'd argue that the optimistic prediction is saying that it takes longer. Because it will definitely stir up the whole society and economy at the very minimum.

They did not need to dumb it down or cut corners anywhere. The early releases of ChatGPT and Bing Chat showed they literally put unmodified SOTA models in the hand of users with no price tag attached. These AIs were known a long time ago but only to some people, remember how a bunch of billionaires suddenly got concerned about AIs a year or two ago? I bet they got early access to these LLMs. But only by scaling it up, they can explore the deeper depths of these models and discover new emergent abilities and realize actually how much progress they had made. Before people didn't really expect an LLM to play chess and simulate world models. Now they just found out these things are probably closer to AGI than they thought and the progress bar got pushed forward.

Basically my rant is that current progress was made over a long time and people just didn't really realize how far they have come until they opened it up to the public. I would not expect too many surprises in the future on the scale of ChatGPT again. If I am wrong though then we will actually start getting serious candidates for an AGI.

As someone in the field, I largely agree with this take but we're still talking about progress over the course of 5 years or so.

Also, fine-tuning/RLHF considerably advanced the usability of the models by the lay public and hasn't been around for that long.

We just kind of walked straight past the Turing test and nobody cares.

On a side note, I think the idea behind the first part of the phenomenon, i.e. that we think you need intelligence (and probably general intelligence) to do most things that machines cannot is that that is the way we do it. So until we could build machines that could calculate (say multiply and divide) we though you needed actual intelligence because we didn't know any other way and that is how we did it. (I remember that calculating and calculators were an actual example in that lesson.) Same thing for chess. And yes, we were right to say that you don't need general intelligence for neither chess nor calculating because we could come up with relatively simple algorithms.

But people a few years ago started to say this about go. And I do remember that after DeepBlue beating Kasparov everybody was like "yeah, but you need intelligence for go because that's a game with vastly more possible moves". And in a sense this is what we saw, because AlphaGo was indeed a kidn of AI, but still people started saying that you don't need intelligence to win go - as long as you are a machine.

Now people started to make up shit about how GPT is just generating text and that's not intelligent (some people, obviously ignorant laypeople, even say that it's just copy pasting text together and other nonsense). Despite that the freaking thing passes high level exams aimed at people. (Of course, you can say that it's still not a sign of intelligence, because the exams do not measure that, we know people are more or less intelligent, the exam measures knowledge, but the amazing thing here is not the knowledge part but being able to answer the questions aimed at people.)

I'm sure that's true for lots of people but I doubt it's true for Hinton. He seems genuinely surprised by the success of large language models.

Before, it was easy to see how good these things were at generating texts and paying attention to text based tasks. But what made them a topic in AGI discussions is how they generalize beyond text and reaching out to different domains of expression despite never getting trained on those. Things like chess or simulating emotions and manipulations spontaneously happened and it wasn't until now that we have documented such events. Nobody saw or expected that.

I don't think this is true - at least not for people in the know in the last 3-4 years or so.

Was it just corporate secrets or they don't want bad press? The Lemoine incident at Google and how Bing Chat turned into an obsessive lover made me think even those who worked with these AIs didn't really consider the full capabilities of their systems.

In some sense, the AI is still 'just doing what it was trained to do' in that it is 'just' predicting the next word. All examples of AI doing impressive behavior boil down to AI doing what it was trained to do (pick the next word [delta some RLHF tuning]), very well.

If you mean complex behavior arising out of what seems like a very simple unsupervised learning task, then this behavior has been known (although not to this scale) for a while.

For example, I distinctly remember being in my 2018 grad class on deep NLP and having a guest lecturer (Alec Radford) from OpenAI and they were demonstrating how their model got SOTA on summarization tasks just by taking the original text and appending the word "tl;dr" and using what the model produced after that token as the summary. It wasn't trained on a supervised summarization task, it just learned it incidentally from its unsupervised task.

The stuff we are observing is in the same vein as this, just even more impressive. But it is not unknown/completely unexpected behavior prior to ChatGPT.

Certainly it was already well known prior to this paper [0], which puts a lower limit on the timeline as at least 3 years ago.

> Was it just corporate secrets or they don't want bad press?

No, I think it was known & published about, although not as impressive as the most recent iterations. The press didn't realize this was a topic that interested people until 2022 (and really 2023).

[0]: https://arxiv.org/abs/2005.14165

Also given enough time anything on that distribution curve will come to pass.

There's also the ineffable outcomes you have to worry about too, not just an intelligence explosion.

If it's possible to "accidentally" build an AI which can "accidentally" hack out of or into secure systems, steal the nuclear launch codes, and subjugate an entire country, then it's also possible to train one to do those things on purpose. If security vulnerabilities exist that make it possible to do those things, then exploitation of them is in no way tied to building any kind of "AGI system" at all, and we can assume many people would be interested in such things right now, before this critical AGI point has been reached.

The solution is to be proactive and determine what vulnerabilities exist that a hypothetical rogue AI (or, more likely, a deliberately weaponized AI) could take advantage of, by training models that are increasingly capable of detecting such vulnerabilities and hardening systems against them. Rather than following the suggestions of the "AI pausers," who aim to prevent AI models from developing such capabilities emergently, this would require acceleration of the development of AI systems that are deliberately designed to do these things, so we can patch whatever vulnerabilities we find, and increase our chances of survival in the scenario that Bing attempts to take over the world in response to a prompt about paperclips.

The alternative, which is "pausing AI", is essentially security by obscurity; we want to prevent people from developing what is essentially a field of math because it could lead to the exposure of latent security vulnerabilities.

Your comment seems to be talking about existential risk, which is not even what the GP was asking about.

If some AI had access to the missile launch system, the best course of action for it would probably not be to launch immediately. This is because nowadays it is very unlikely that it would be able to repair itself so launching immediately would ensure its own destruction (and probably auto-destruction is not its goal)

If it was discovered it could just threaten humans with launch if they do not help it reach the state at which it would be able to repair itself (at which point humans would no longer be necessary)

What you and a large majority of everyone else don't get is this,

AI does not need to become AGI to destroy us all. The minimal much easier to reach requirement it needs to do that is make it so people can't find jobs.

People will do the rest in accordance with fundamental societal mechanics that have been known for centuries.

Economies work in a loop. You trade time for production (labor) in the factor markets, and you store it in money, you use money to buy what you need (food).

Companies buy that labor to produce goods which they then go and sell to fulfill those basic needs.

What happens when basic needs cannot be met, because companies don't buy the labor, and government has a history over the past generation (which is commonly 20 years) of doing largely nothing.

When the people starve, there will be blood in the streets, and there are a lot more of them than there are people at the ideal corporation which would be 1 person at the top with robot slaves replacing everyone else.

The only solution would be a central structure that provided for basic needs without a market. This has been tried many times over the past 100 years, all of them failed (afaik). You run into what's known as the economic calculation problem (or debate) depending on where you look, and a whole host of other difficulties which have no solution.

It works for a time, then something unpredictable happens which causes shortages, then those shortages causes death on a massive scale. There's a lot of written history showing this.

It reminds me of Y2K hysteria.

When old jobs die, new jobs arise. That has always been the case, and I haven't seen any evidence of that ending.

It also ignores a growing body of evidence.

If new jobs arise as you say in the same amounts, why is it that young people are having such a hard time finding gainful employment these days?

Take a look at the statistics, you'll find as some already have, that the job pool has been shrinking, and older workers who have more experience out-compete younger workers for entry level jobs because the cost to train (is high), and production is lower in workers with no experience than those with experience.

The only indicators we would have that this is happening, just like inflation, are lagging indicators.

By the time you know it is happening, its then too late to do anything. That's why this problem will probably kill us all. We as a species historically do not handle cascade failures well because we don't perceive the danger until its too late.

Its a problem where our natural biases work against us. Its normal to just discount problems we have no context, which is why the out-of-context problem is so dangerous.

Do you think the native people's of the Carribean thought there was danger from white sails of Columbus ships?

By what measure? Youth unemployment is right now around its lowest level in…quite a long time. [0]

[0] https://tradingeconomics.com/united-states/youth-unemploymen...

BLS data has been manipulated and methodology changed to always paint the rosiest picture for policymakers.

The shadowstats website has continued with the older more accurate methodology where it shows a more realistic view, when it can.

US data for unemployment has been fundamentally flawed for decades. In essence, the methodology stops counting people looking for work when they have been unable to find work after some amount of time. If you aren't actively looking for work or recently had work, you aren't counted. Actively looking doesn't mean interviews, it means being registered in a work program, which has a large number of requirements and competition.

The resolution foundation has several good analysis of multiple countries that show this trend.

If you want more specific details than that, I'd suggest "The Pinch" by Willett. It has many credible references, supporting this at the back (too many to list).

Where other similarly themed books like "The Theft of a Decade", offer no more than opinion, this covers what you are looking for with gratuitous references.

There is nothing I can do about that.

The problem is that humans aren't getting specially smarter as generations go, but technology continue to grow

How tho? Will LLMs replace grocery store workers? Farmers? Carpenters? Plumbers? Electricians? Truck drivers? Train drivers? Chefs? Cleaners?

They won't even "replace" middle management. Yes, they can help them write emails, plan stuff, research it and do all the fluff boilerplate parts of the work. Can it communicate with people properly, understand when somebody is overestimating, underestimating, lying, not lying, when the worker is actually sick or just pretending?

In the twitter-hn echo chamber, it seems like an LLM can do it all since it can "write text and code", but literally nobody doing any work outside of "we write text" sphere isn't really worried. Hell, as a developer I ain't even blinking - I've worked with GPT for more than a year and am well aware of it's limitations - and yes, some can be solved by chaining another instance to check the instance and then another instance to whip the instance into submission, but that is just unproductive and way more costly to execute than just to have a human expert do it.

Also, innovations that increase productivity don't lead to losing jobs, they lead to increased productivity and economic value. Otherwise, our civilisation would be starving since the wheel. It's more likely we will slowly progress towards the "universal income" utopia by using NGAI (near-general AI) than we will devolve into a bloody chaotic hungry mess.

> because companies don't buy the labor But who will buy those companies products? What do you define as labor? As far as I know, OpenAI isn't building highly advanced robots with multi-sensory functionality that are as robust and as adaptable as humans. If they were, I'd be a bit more afraid. Just a bit.

It can happen any number of different ways, and you won't know which way until its too late to do anything about it. The danger of cascade failures is you don't perceive an issue until it is too late to do anything about it, most times.

The rate of change is very important. ~10 years ago, object tracking and detection couldn't detect objects very accurately. This was before resnet. Then resnet came out and it changed everything. Then NLPs, Then Visual Transformers. 3dimensional depth mapping, GANs and now LLMs. Its progressive and exponential. No one could predict how soon the next breakthrough would be but they fell like dominoes so far at an increasing rate.

You don't need middle management if you have no low level employees. Self-service kiosks are already being used in a lot of countries to replace workers. 3D printing is replacing other areas. They already have cleaning robots. Robots are fixed cost assets. People's expertise are recurring cost assets. Its clear which is the better investment from a business perspective short-term.

Anyone that isn't really worried about these things is going to be blindsided because they fundamentally don't understand the nature of the problem. Imagine a Code-GPT comes out in a few months that reads a requirements specification written in regular language and outputs code you'd normally have spent months writing in a few seconds. No expertise needed. You won't know ahead of time. Someone does it, publishes it, your entire profession is mostly out of a job.

If multiple chains cost cents, how much chaining needs to happen for it to equal minimum wage, let alone a mid-level salary?

You are wrong about your statement that innovations that increase productivity don't lead to losing jobs. They do when those productivity gains exceeds demand. Demand is constrained by the population/market size.

While I'm not too worried about losing my job since I tend to do high IQ design work, those low and mid-IQ jobs normally filled by average or above average intelligence people will be hit the hardest.

You should be afraid, simply because the rate of change exceeds our ability to react in any meaningful way. If there is a problem, we'll never be able to see it coming until it is too late. Also, when has there never been some problem.

If you've done any significant study on socialism, you'd know both market and non-market socialism suffer from the economic calculation problem. Its not solved, people have been working on it for over a hundred years. Shortages cause failures which then cause death on a large scale. Universal Income is equivalent to non-market socialism where resources are simply moved around internally. There is no actual price discovery, prices deviate so much from the rational cost to produce those items that you run into characteristic issues of this problem.

Economic value by itself offers little. It can be solely concentrated in one person, or diffused throughout an entire country. One has benefit the other doesn't, but in terms of economic value they are the same. Its a made up number that doesn't measure what it claims to.

All of which comes from an era without the public internet, the smartphone, or AI.

A famous catchphrase, this time it will be different. Hope is not a strategy.

If we're trading catchphrases, here's one I prefer:

"In times of change learners inherit the earth while the learned find themselves beautifully equipped to deal with a world that no longer exists." - Eric Hoffer.

That said, I'd prefer it if someone addressed the point, which is that the analysis of what's possible dates from an earlier technological era.

If the contention is that technology doesn't change what's possible in society and that society will always be the way it is was for most of the 20th century, I think we can safely ignore that.

The contention is if history has gone through multiple and innumerable cycles and has ended up at the same outcome multiple times before, given an infinite number of potential outcomes, there must be common principles that led to the clustering at that outcome.

If you choose to ignore what those conveniently documented outcomes were and by extension what led to them, can you exercise any reasonable control over something you have no knowledge of?

Would the statement, "This time will be different, have any credibility if you have no control or knowledge of the issues?

Everyone at some point will say anyone, anything, and nothing, and what is said will likely be less true than not true when it is not backed with support.

Rational and rigorous approaches from first principles lend credibility to something being potentially more true than not, and by the law of approximation we can improve these approches over time until we eventually get to the point of it being true.

Wouldn't you say its better to be true, than not true if you want to succeed and live?

The first problem is that this is simply a false premise. Very few people would make such a claim, and none can back it up with evidence.

The second is that you still seem to think that technological change has no impact on history.

To claim otherwise is simply discounting a large body of scientific work which is highly credible and supported, how can you claim doing so is credible in any way?

The second is just putting words in my mouth, I have no idea how you could possibly come to that conclusion about what I think based off our short conversation here.

I never said it, I never inferred it, it looks to me simply like a psychological projection you've made of something you think, Largely because I don't believe that in the slightest. Not in the slightest, and no rational person would.

So can you clarify 'specifically' what I said that made you think that?

Did these past catastrophes involve AI and computers?

This line is at the very center of why we will eventually drive ourselves to extinction. I agree that it is not possible to magically press some pause button, but neither does this mean that the concerns are unfounded.

It could be 100% true that pausing gives OpenAI a competitive advantage, and it would have zero bearing on the actual or imagined risks that such a pause would or wouldn't avert.

Almost every time this subject comes up, everyone involved falls into the trap/fallacy of binary thought and false dichotomy. As soon as the "anti" crowd speaks, the "pro" crowd moves into refutation mode. As soon as the "pro" crowd speaks and calls attention to all of the possible benefit, the "anti" crowd falls back to the same base position.

While we argue about this infinitely complex issue in binary terms, we're bringing about the very thing that concerns everyone. Whether or not AI will eventually destroy the world, we keep forgetting to actually figure out if this is a real concern, and instead argue about the economic pros and cons and inevitable competitive advantage that someone might gain, as if that's somehow important to the underlying issue.

All of which is to entirely miss the point. As long as we continue to argue about why AI is good or bad, and why a pause is impossible or undesirable, we continue to miss the point that it's both good and bad, a pause probably is impossible, but that doesn't in any way remove or mitigate the concerns behind the call for a pause, or reduce the validity of those concerns.

We need to have an intellectually honest and nuanced conversation about this problem, and while such a conversation should acknowledge the reality that a capitalist environment plays, it should also acknowledge that the concerns transcend that reality. And it should acknowledge that when idealistic positions such as "pause for 6 months" turn out to be impossible, that just means the problem is harder to solve, not that the underlying issue doesn't exist.

I’m baffled as to why this talking point keeps getting repeated so much.

Why should I care if OpenAI’s competitors get an advantage? Why do you care? Do you work for OpenAI?

But even still, I don't understand where this comes from. The 6 month pause would also apply to OpenAI. I thought the idea was to give legislature and governance a chance to catch up

   > The whole "pause AI or it will destroy the world thing" is just
   > a large pot of hype anyways, pushed by journalists lacking the
   > understanding of technology, twitter-bros who are building "AI
   > copywriters" and "profile photo generators" and owners of "AI"
   > companies.

- Yoshua Bengio

- Elon Musk

- Steve Wozniak

- John Hopfield

- Gary Marcus

- Jaan Tallinn, Skype co-founder

- Evan Sharp, Pinterest co-founder

- Chris Larsen, Ripple co-founder

The technical nature of the problem is a tiny piece of the claimed impact, and an even smaller piece of the question of whether the proposed policy would, in fact, resolve any real problem and do so without causing greater harms than it prevents.

I think it's a bit silly to list founder/CEO-types and take their opinions as if they know the topic well, unless they have the appropriate AI-related background.

From what I've read, Yann LeCun himself allegedly "signed" the letter. His name was removed from the list after he issued a public statement denying he'd signed, and in fact actually disagreed with it.

Which one is there the strongest case for trusting their judgement on this issue and what is that case?

(And are these actual verified signatories, because the letter had a problem with false signatures?)

I guess crazy things like that happen when you consider the content of what a person says, rather than their cultural status or designated authority.

>- Elon Musk

the quintessential twitter-bro

>- Steve Wozniak

currently fronts 'eco-friendly' crypto scam

>Jaan Tallinn, Skype co-founder

"Tallinn participates in the effective altruism". Somehow crypto bros love that movement. Tallinn even invested together with SBF in various "ventures", some directly competing with OpenAI (Anthropic).

Now: Bengio talks about the reasons he signed it and the reason is more in line of "machines pretending to be humans" i.e. everything from disinformation, security issues to companies interjecting LLMs where a human should be in the loop, causing all kinds of problems.

Elon Musk is literally a twitter-bro that has shown his lack of technological understanding more times than I can count. Also his "visions" are nothing but badly regurgitated sci-fi tropes, distilled and unrealistic, sprinkled with the power of money on top targeted to people who don't understand technology.

Marcus is constantly saying that the technology is hallucinating and too-early to be deployed.

Hopfield I have no qualms with.

But what do Woz, Jaan, Evan and Chris have to do with AI development?

The infosec world figured this out ages ago. Yes, it's possible that some bad actors will gain access to capabilities that they didn't have before - but the strength of a billion eyes is huge, and can rapidly accelerate solutions to deficits.

What we need is a massive, open, community curated dataset that continually evolves. Craft regulations that enforce data availability and free access. Without such mechanisms, AI systems all have a SPOF that's just waiting to bite the collective us in the ass.

The main barrier to entry for them is the massive amount of refinement required to make any useful amount of the required materials, like multiple warehouses size operation.

---

The big difference here is stable diffusion has shown you can train a highly capable model under 1 million dollars. LLaMa is making some interesting advances as well (finetuning a LOrA takes ~3 hours on a 4090). Training/finetuning/inference capable AI is quickly becoming more feasible (albiet trailing the commercial stuff) for the average consumer hardware.

Can growth be too fast?

- development behind walled gardens and closed doors, which will accelerate the wealth gap to war- and poverty- inducing levels

- access to this powerful game changer for the people who would otherwise be losing their purpose in the social structure. There's no reason your job has to evaporate as an artist, or programmer, or anything else if you can just keep doing the same thing empowered by a cheap, accessible force multiplier like AI.

It really comes down to whether or not the closed models are _enough better_ that they can start charging exuberant rents to people so they can effectively continue contributing to society.

The paths I see, in broad strokes:

- models become open and easy to hack on. We all lose our jobs. But we can now find new jobs using these models to do similar work.

- models stay closed and only get developed by BigCo. We all lose our jobs. Remediation for this development may or may not happen and who knows what it will look like.

The core difference I think you and I are dancing around here is a belief in our collective ability to hit the brakes on progress. I understand the desire to do so, but you won't stop a river by putting a rock in the middle and I think this is a situation with that level of momentum.

We are on the cusp of the next step function in humanities capabilities as a technological civilization. The upsides are enormous and I welcome them, but the downsides are even bigger and we should map this out before running off the damn cliff. Technologically, politically, economically.

The fact that it would be hard, or difficult should not play into whether we have the discussion. This is the issue and reasoning that proliferated the bomb. The bomb didn't have to be dropped on Nagasaki and Hiroshima and unfettered bigai doesn't have to enter the winner take all capital arena.

Even with just chatgpt4 level performance, millions of people are in the process of losing their jobs. I had to argue at a bar a couple weeks ago for a dev manager not to lay off his entire team of green junior data engineers. He claimed he could work along and replace the three people that reported to him... I not only saved those peoples jobs (hopefully) but also his own. There is no way he could have stayed afloat even managing requirements.

Huge changes are coming and we aren't prepared. I'd rather have a 15 mph collision than a 55 mph one.

I am not convinced that two AI researchers are qualified about the larger ramifications of their creations. They don't have a grounding all of the other skills necessary, are they politicians, economists, psychologists or philosophers?

>> Even with just chatgpt4 level performance, millions of people are in the process of losing their jobs

What jobs specifically, in which sectors, and what aspects of the role (or is it the entire one) will be replaced by "AI"?

Also, who is implementing all these changes? Who will run the "AI", who will pay for it?

https://en.wikipedia.org/wiki/Sealioning

For reference, my debate approach was not sealioning. Given the extremely bold claims made by OP, I wanted to know if they had consider ed alternative views as their comment read as unrealistic "hype".

That doesn't seem like a fitting comparison, considering it's a highly debated topic to this day.

> I had to argue at a bar a couple weeks ago for a dev manager not to lay off his entire team of green junior data engineers.

This has more to do with that particular manager not understanding the current state of AI, and buying into the hype, than the technology reaching the point where developers are being made redundant by AI.

That manager would've come to a swift realization of his mistake very quickly. So you managed to burst his bubble of delusion, but this is not a sign of AI taking over anytime soon.

> That doesn't seem like a fitting comparison, considering it's a highly debated topic to this day.

A comparison can be apt independent of how inciteful the statement is. The dropping of the bomb was a showcase of force, it didn't have to happen on a population of people. The demonstration could have happened on Mt Fuji, or Okinawa.

> These cities were largely untouched during the nightly bombing raids, and the Army Air Forces agreed to leave them off the target list so accurate assessment of the damage caused by the atomic bombs could be made.

> The Target Committee stated that "It was agreed that psychological factors in the target selection were of great importance. Two aspects of this are (1) obtaining the greatest psychological effect against Japan and (2) making the initial use sufficiently spectacular for the importance of the weapon to be internationally recognized when publicity on it is released. ... Kyoto has the advantage of the people being more highly intelligent and hence better able to appreciate the significance of the weapon.

Fucking immoral human experimentation.

Ok, back to AI. We are on the same level of change, before and after nuclear manipulation, before and after AI. We don't need to be demonstrating on the economy, we have an opportunity to show this technology in isolation before just unleashing it on the world in a grotesque tech demo.

The comparison fits.

Now what do you suggest next?

I predict we will have distributed training systems soon on peer to peer like network for collaboratively building free models to use, if we can all pitch in some GPUs we can get 100s of thousand of individuals working on training giant LLM models together. It’s an unsolved problem currently but should have a solution soon.

Additionally today we can build sub-$10K systems to do self hosted inference on 65B weights models.

I’m seriously confused why anybody thinks anyone will even remotely care about any kind of pause is asked for.

For sure everyone with computers and a reasonable level of skills are now looking at this field. The progress will be exponential and uncontrollable in my opinion. I’m fine with that and I also think there is absolutely nothing that can stop it anyway.

I can say that Google and Microsoft would definitely care, just as two specific instances of "anybody" that I have worked for and seen what their legal review process gating new product launches is like.

You seem much more optimistic that the community efforts are <6 months behind those companies than I am; I would be happy to be wrong here and see an open-source self-hosted LLM with better than GPT4 performance by September.

I think people underestimate how many entities have the ability to easily acquire that amount of resources. It’s less than $200M to purchase that amount of hardware. Militaries spend $2-4M per missile. This is nothing for a lot of organizations.

We already have self hosted LLM inference that reaches gpt 3.5 turbo level.

Google and Microsoft are notoriously slow moving and probably less relevant than smaller faster operations like OpenAI.

Yep so global totalitarianism is the clear answer. Makes sense.

I’m sure nobody will resist.

After the 6-month pause is over, then what? Another 6 months? What is to be hoped to be achieved in those initial 6 months other than non-participating entities getting a leg up? If there is no guarantee that the 6-month pause will not become one year or a few years, then maybe some researchers will move to non-participating organizations/states in anticipation of an extended pause.

Even on the individual level as for me the discussion of such ideas has me thinking individually if I should take any defensive steps or not.

I guess you could try comparing capabilities, but without access to the training data we have basically no effective way of running benchmarks against the GPT-4 api, even if you are one of the lucky few who already has API access.

Here, Yann LeCun is saying that we don't have systems that can reason and plan.

That needs a qualifier: we don't have _statistical machine learning_ systems that can reason and plan. There are plenty of systems that can reason, and plenty of systems that can plan, and even some that can do both, but those systems are not statistical machine learning systems. Rather they are what is derisively dismissed as "Good Old-Fashioned, AI": they are classical, logic-based, symbolic systems, automated theorem provers and planners.

Reasoning, in particular, specifically deductive reasoning, is solved to a degree that cannot be surpassed: the Resolution principle is a sound and complete deductive reasoning system with a single inference rule that is easily run on a computer because of its simplicity, and because it is a single inference rule; while other sound and complete systems for deductive reasoning exist, they do not consist of a single rule and a human must be on hand to select the appropriate rule at each step of a proof. Or, they are just extremely expensive to run whereas Resolution, thanks to its One Simple Trick™ of unification can be executed efficiently.

As to planing, we have fast algorithms for planning today, as for all kinds of tree search, constraint optimisation, SAT-solving and the like, and those algorithms are routinely used in the industry; except of course they are no longer recognised as "artificial intelligence" because they are so common. The so-called "AI effect".

In any case, reasoning and planning, and many other tasks that were perfectly possible with classical AI are, for the time being, impossible with deep neural networks, as Yann LeCun (and not just anybody) says. We have regressed. In our passion to build ever better classifiers, we threw away the ability to reason.

They argue to regulate application not basic research.

If they want pause, they should first convience all companies bigger than OpenAI (Google, Tesla, Meta, and the Chinese counterparts) to pause all deep learning model training, to keep the competition fair.

The worst thing that could happen is pausing OpenAI, big companies catching up, and destroying a smaller company by having more cash.

Also noticed something: The people who believe consciousness and intelligence and creativity and will are wholly contained in mechanistic wetware (which, in all honesty, should be the null hypothesis) are the most afraid of AGI. Dualists and other people who believe that consciousness resides in some soul-like thing seem much less afraid of this.

One thing’s for sure: The smarter it gets (or seems), the more that any difference between “machine intelligence” and “human intelligence” will become apparent.

That seems unlikely. We usually ascribe the adjective "smart" to animals and/or AIs whose behavior resembles human behavior in some way. So, if anything, the smarter an AI gets, the more closely it will mimic human intelligence – at least as a general rule of thumb. That's not to say there will not be differences between human & machine intelligence but essentially we're optimizing AI against a loss function which rewards human-like behavior.

Like, there is a good argument to be made that believing that consciousness, intelligence, etc are distinct, enumerable things "contained" within whatever system you choose in fact makes you much more of a Cartesian than any of the details around soul and body. Presenting even the dilemma this way shows you haven't left the enlightenment at all!

Just if you are interested in this stuff, I would check out what people have thought about since the 1600s. The more you feel like these things are already solved, the more it shows the poverty of your current conceptions around them.

"Descartes denied that animals had reason or intelligence. He argued that animals did not lack sensations or perceptions, but these could be explained mechanistically. Whereas humans had a soul, or mind, and were able to feel pain and anxiety, animals by virtue of not having a soul could not feel pain or anxiety. If animals showed signs of distress then this was to protect the body from damage, but the innate state needed for them to suffer was absent. Although Descartes' views were not universally accepted, they became prominent in Europe and North America, allowing humans to treat animals with impunity. The view that animals were quite separate from humanity and merely machines allowed for the maltreatment of animals, and was sanctioned in law and societal norms until the middle of the 19th century."

Not sure where the GP suggested these questions were "solved."

Consciousness, while philosophically interesting, doesn't necessarily correlate with the risk a general intelligences causing harm.

Enslaving humans is pretty universally considered wrong, even when it was a legal practice, but nobody sheds a tear about the cow they had for lunch.

Amazing, eh? That’s why beliefs matter.

https://www.nytimes.com/2014/12/12/world/europe/dogs-in-heav...

And as a matter of experience, the vast majority of people interact with the world dualistically regardless of their metaphysical beliefs or lack thereof. It is possible to both recognize the illusion exists while remaining very much under its spell as a matter of experience and as a frame of reference.

This framing becomes problematic when one continues to base assumptions about intelligence and language on the felt experience of “I”, regardless of the seriousness of the metaphysical position.

Think AI judges. They'll never be sold as such, but the impact of their operation on individuals may be just as impactful.

I'm still catching up on my reading, but the existential risk of AGI as I understand it is chiefly that we don't know how to align AI (and keep it aligned) to be beneficial/benevolent for humanity, but the risk is contingent also on a few other things things - that we develop AI that agentic and able to develop and pursue it's own goals and sub-goals, and that it has power-seeking or game-theory strategic awareness/planning. The extra dangerous situation is if it is better than humans at making better AGIs and allowed to improve itself (or build its successors).

Regardless of when we reach AGI, I think the alignment issue is a key one for both AI ethics and safety. As for the rest, while they aren't necessarily things that will happen, since it seems to the first thing people most want to build - the other day, the top three projects on https://github.com/trending were all projects to try to make ChatGPT autonomous/agentic.

FWIW, I don't think the 6-month ban is a good idea, but I also think there's a lot of xrisk w/ AGI and it's better to be thinking about it seriously sooner, rather than later. What I've been reading:

Ngo, Richard, Lawrence Chan, and Sören Mindermann. “The Alignment Problem from a Deep Learning Perspective.” arXiv, February 22, 2023. http://arxiv.org/abs/2209.00626.

Carlsmith, Joseph. “Is Power-Seeking AI an Existential Risk?” arXiv, June 16, 2022. http://arxiv.org/abs/2206.13353.

Yudkowsky, Eliezer. “AGI Ruin: A List of Lethalities.” Accessed April 8, 2023. https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-r.... June 6, 2022.

paulfchristiano. “What Failure Looks Like.” Accessed April 6, 2023. https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-.... March 18, 2019.

Soares, Nate. “The Value Learning Problem.” In Artificial Intelligence Safety and Security, edited by Roman V. Yampolskiy, 1st ed., 89–97. First edition. | Boca Raton, FL : CRC Press/Taylor & Francis Group, 2018.: Chapman and Hall/CRC, 2018. https://doi.org/10.1201/9781351251389-7.

Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.” arXiv, July 25, 2016. http://arxiv.org/abs/1606.06565.

> If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter.

> There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all.There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all.

It is a valid criticism that Yudkowsky and others are over-staging the abilities of current day GPT. For example Eliezer says "can it reason? It can play chess". But I looked at a GPT chess game (actually published in an article on his less wrong site). GPT has no idea how to play chess. It can make some moves in the opening that match opening theory which it has been trained on. But then it will lose a piece because it has no actual ability to think 2 moves ahead. And then when it gets into the end game it tries to make illegal king moves.

As impressive as GPT is, this does not seem to be an AGI that can properly understand and reason, and I am even skeptical that the current LLM models will lead to this. We have many years before AGI wipes us out :) I really don't know anything about AI though and we never know when the next breakthrough advance will occur.

[1] https://www.youtube.com/watch?v=AaTRHFaaPG8 [2] https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

I can't help but think a lot of humans haven't achieved the "human level intelligence," these speakers attribute to everyone else. They talk about these models not being able to achieve human level intelligence (HLI?) because even though we are language oriented, we can do things like learn to drive in a few hours vs. this task being currently much more difficult for machines - but all those comparisons are to competencies in our physical environment that are not relevant to a media environment, which exists as a substrate for the narratives people use to shape their beliefs and identities. If we overestimate how intelligent people really are, LLM's may be more powerful than we think.

Explosives may be a useful analogy, where we could bring to bear mechanical power and leverage to a ceiling of its ability in our environment, but once we harnessed chemical (and eventually, atomic) reactions, the things we can create with them can be trillions of times as powerful as our respective abilities. Humans with springs and levers weren't that powerful after all, and the risk is that what we call HLI is, relatively, springs and levers level powerful. This might seem like an argument in favour of a pause, where in LLM's we have managed to invent the equivalent of an intellectual explosive, but I would argue we need to accelerate and spread understanding of ML past the point of where it can be monopolized by a small cadre of social engineers.

The only thing that prevents you being managed by a machine is the physical competency and market value to decline it, and pausing to add governance now will be specifically to deprive you of that. The advocates for an ML development pause seem all about progress when that means redistributing things others already have and inserting themselves as governance, but they are absolutely against it when this progress means producing something net-new that could reduce the ability of said managers to be the ones to be the redistributors.

I think we should only accelerate AI tools development so that they can provide more equal opportunity to all, and let people create their own disincentives for using it for the systemic oppression that I can also guarantee it will be used for if we pause to let the gatekeepers in.

For now ai is not science or research or innovation, it's just money spent so that companies can say they're at the bleeding edge.

I think there might be a reference to it in this article. Also, linking to a section talking about this proposal, since that might answer other questions you have.

https://thezvi.substack.com/p/ai-6-agents-of-change#%C2%A7ad...

https://aiimpacts.org/2022-expert-survey-on-progress-in-ai

2. Ng was Chief Scientist at Baidu, one of the companies involved in implementing mass surveillance in China, you can ask him.