Wednesday, January 31, 2018

How to Bring Delphi Back from Life Support

I was recently commenting on Facebook and asked what could the community do to help Delphi get more popular again.  My answer? Uh Nothing. 

A real change would start with the product actually getting a serious refresh, something probably that nobody can afford to do, and which EmbarcIdera is extremely unlikely to fund.

Cue up a nostalgic 80s song, maybe Glory Days, by Bruce Springsteen.  Grab a beverage. I'm about to rant.

The Glory Days of Delphi were from the Delphi 5 era until Delphi 2005, where the product foundered severely. 2005 was so bad, and 2006 was only slightly better, that few people could use it. To this day I encounter companies still using Delphi 5 or 6 or 7 because of this ancient developer and QA failure made in the tail end of the Borland era.   Let's face it. Shipping Delphi 2005 was the worst thing that the management ever did to the product's reputation. Even worse than the well known shenanigans at their Sales level where they were pushing their (now dead) Java IDEs on their existing Delphi accounts.

The blowback from the instability of 2005 was actually less than the blowback for the UI changes. Developers (it me!) are a surly bunch.  You moved my buttons. Damn you!

Now the UI changes were well intentioned, and well done, and were intended to bring Delphi out of the Win3.1 era UI of Delphi 5 through 7, and into a docking style IDE, comparable with Visual Studio 2005. I like the new UI in 2005+ delphi editions better than the old UI, but the point is that the old IDE was more stable.

There has not YET been another full refresh of the IDE.   Delphi has continued to be usable, and a good productive tool, and while it's IDE architecture continues to look worse by comparison with its one time equals, it is now so far behind it's not even funny.  Remember there are people using FoxPro and Visual Basic 6 IDEs and loading them up in Windows 10.  We're more like them, than different, at the moment.  The registry key where delphi gets installed gets a regular major version bump, we're at what 18.0 now?  But the IDE architecture is frozen in time.



The bds.exe code of 2018 is mostly the same in operation, and contains most of the same assemblies and parts (binary dlls) as the bds code of 2005, with the addition of creaky refactoring code built in J# (yes JSharp, that no longer supported .net 2.0 language), error insight that doesn't work (due to having its own parser which doesn't parse all the language features, or even the compiler search path behaviour of the real compiler). The editor core in C++ is buggy to this day and crashes, and loses people's work. 

A lot of talented people worked really hard and fixed a lot of bugs, and Delphi XE8 and up, including the current Delphi 10.x series are very stable IDEs. Don't get me wrong. I appreciate that work.

But what we have here is a polished antique.  The editor core is basically the code from Borland C++ for Windows' editor, from about 2001.  The code is a legacy codebase, and it needs a total IDE overhaul.

The delphi DCC32 and DCC64 compilers are relatively good compilers, excepting that there is no significant OPTIMIZATION available for DCC64 still.   If you are familiar with the level of optimization work done in modern C++, and you compare it with Delphi in 2018, you're gonna be sad.  I think that the Nextgen picture may be significantly more rosy with LLVM's optimizers at work, but that's never going to be an option when Nextgen was designed from the ground up to contain many breaking changes to your code, including string handling, and other fundamentals.  That's not gonna be pretty.

But the Compilers, classic and nextgen, produce really fast code, even though they are not at parity with say, the Visual C++ 2017 optimizing compilers.    I'm actually fine with that. I actually still kind of  hate C++.  Pascal is a beautiful language.

The bigger problem is the IDE is not very good. 

Spend 16 full hours getting work done (pair programming) with a colleague who knows what they are doing in Visual Studio (perhaps with resharper or not), or IntelliJ IDEA and you will realize that what we mean when we say "IDE" in delphi is at a point similar to what Most IDEs Were Capable of Circa 2005.

Alright Smart-Ass What would You Do Differently?

Fair question.  If you gave me a 6 million dollar development budget and two years I would rearchitect the IDE, but not the compiler, or the language specification:


1. Subprocesses are key. Yes, yes it will use more memory. It will crash less. Are you sold yet?

1.1. Make the editor core a subprocess (RADStudioEditor.exe). Less crashing.  Large file search/replace should even be able to happen in background in parallel.

1.2. Make the compiler and code completion engine a subprocess (RADStudioLanguageServicesEngine.exe).  Kibitzing compilers, code completion and error insight compilers would share a single tokenizer/lexer and grammar.

1.3. Make the IDE main window (shell) a lightweight shell written in Delphi (RadStudio.exe).
4. Make the form designer also a subprocess (RADStudioFormDesign.exe). This subprocess would be capable of loading a set of BPLs needed to open the form.

2. First do no harm. Don't Tread On my DFMs. I would fix the ancient frustration caused by failure to locate and install all BPLs that a form requires before you open a form.  It would be possible to open a form without the damn thing DELETING YOUR COMPONENTS.   Similarly, all the places where it asks you sixty million times in a row are you sure you want to delete your components would default to no, don't bug the user, and don't do destructive stuff, instead fall back gracefully and do nothing.  Those fun times when you have to open task manager and kill delphi to get it to stop asking you 9 million questions. Good times.

3. Stability is key. When a BPL fails to load or segfaults, instead of the IDE going down, you would lose one DFM editor tab.  

4. Modern editor features.  Delphi's next IDE needs navigation and quick look features on par with IntellIJ IDEA and Visual Studio 2017.

5. Unified compiler and code-completion (kibitzing) engines, with a single root grammar and Lexer.   The compiler and the code insight engine need to work similarly to the modern architectures used in Visual Studio and Visual Studio Code.

And a few things I would NOT do:

1. We Don't Need No Stinking New Language Keywords because We Are Not Jealous of C++.  The language in Delphi is actually it's most important asset. Adding crap to the language is not gonna make it better. Delphi is kind of beautiful and clean, like Google Go.  I actually think the syntax of anonymous methods is the ugliest part of the current language spec. I'd like to stop now before we end up with crap like they have to deal with in the latest C++ language spec.  Delphi's module system, and runtime plus compile time typing system are better, in my opinion, than C++'s mostly compile-time-only type systems, and current lack (in 2017) of modules. I don't want to copy C# either.


2. Break stuff. I would not break anyone's code, DFMs, .pas files, etc.  Absolute compatibility is the only way to guarantee you can upgrade.  The decisions already made to break stuff in nextgen, while I understand the difficulty that lead to the reluctant change, I still thing are going to be the undoing of the entire thing.



Would 6 Million American Dollars be enough budget to do all the above? Would it take 2 years? That I don't know.  I just know it would take a lot of work.     And given that the business case for the above might be that the payoff would be around $4 million dollars, it remains probable and possible that doing all the above is a business mistake.  

 What I do know is that the above is the list of things being NOT fixed, are among the things that would make most young developers who know any other language, and even most old farts like me, want to Stay The Hell Away from the product.

When Delphi works, great, it's a joy to use.  A proactive solution to the long term trouble with Delphi stability requires a modern multi process architecture, and a significant R&D commitment.  It would also be nice if you still had some key developers who knew where all the bones are buried.   If you didn't have all that, I'm just saying, that maybe six million dollars is still not enough to do the work above.

The market realities today are stark. You can get a trial of IntellIJ IDEA for free and run it for months, and you only pay a few hundred dollars for it if you buy it.  Visual Studio Community is freely downloadable and is the default choice for Windows-Centric developers today, which is funny because WinForms and WPF still suck.   

Delphi is a niche product today, an important and interesting one, but it's probably never going to be even 10% as popular as the big boys, ever again.  I still think it's a great thing to build Win32 apps in, and I do wish it wouldn't eat my DFMs and my code on me.

Rant over. [Mic drop]




Postscript:   A very senior Delphi Developer where I work pointed out that another core Delphi IDE design mistake is that the Library Path, and Component Installation Registry keys, and other similar items, exist at all, at a global level, which is not committable to source control, but which does affect your ability to open, share, and build codebases in Delphi on multiple developer machines, and that these settings should be  per-project and be as files on disk, not in the registry. I agree with him heartily, and feel he was right when he said that is perhaps the single worst problem with the current IDE design.  If I were to replace the existing crap, I would introduce a concept like Workspaces in most java IDEs, such as  IntelliJ IDEA, to the Delphi world.  A copy of the files and their workspace should be all you need to transport and build Delphi code anywhere.  And it should be possible to load different versions of different components into different projects without the egregious workaround of bds.exe -Rsomething




32 comments:

  1. Why not use Intellij, improve the i-pascal plugin, write a DFM editor and a debugger. That would reduce the amount of work a lot.

    The ErrorInsight parser isn't that bad but it suffers from race conditions.

    ReplyDelete
    Replies
    1. A smaller change to errorinsight would be to have it give up and not show false errors on units it can't parse. The errorinsight compiler is worse than useless. And I didn't even mention that the code behind that parser locks up the File and Edit menus when you drop them down, sometimes for up to 60 seconds, while locking up one CPU core, and consuming 800+ megabytes of memory, because of the heinously bad design of the code completion and error insight parsers.

      Delete
    2. I've often wondered if EMBT partnering with JetBrains and letting go of producing its own compiler from scratch would be a good idea. They don't seem to have the resources to support compilers, IDE, two frameworks, database, etc. Another alternative would have been opting to use Qt as their cross-platform framework rather than buying then developing what would become Firemonkey.

      Delete
    3. That should read "IDE from scratch", not "compiler".

      Delete
  2. I disagree. I don't think Delphi is on life support. It is alive and well, and kicking and dancing. No, it does not do everything I'd like it to, but hey, neither do other languages.

    ReplyDelete
    Replies
    1. If Idera's happy with how fast it sells into new accounts, and you're seeing new people showing up in Delphi forums, then you have a point. If the average Delphi developer is 50+ at this point, you do not have a point. Delphi's community (including me) is largely graybeards. The adoption level among young users is pretty abysmal. Delphi has not had a significant refresh to its aging IDE since 2005. Delphi was in critical care from 2005-2008, and has been on life support since then. That's what I mean by "on life support". This is not new, it's been the trend since 2005. Firemonkey was cool, when release, but it's on life support now too. The firemonkey team that built it, is gone, made redundant, laid off. The St. Petersburg office is closed. The Spain offices are closed. The US offices are a shell of their former selves. That's LIFE SUPPORT.

      Delete
    2. Pascal hasn't been using in AP Computer science courses since '98 or '99, is out of academia except for South Africa, commercial publishers gave up on it after 2005, Delphi is vastly more expensive than competing products, it lacks an arm's length list of modern language features (unlike Oxygene, which is like Delphi++), the jobs market is virtually nonexistent, and no one under the age of 40 knows what it is. No multinational company is using it for an infrastructure project (often said to be a sign that a language has achieved success). Delphi has also had 4 owners in 10 years - Borland, CodeGear, Embarcadero, Idera, with each change leading to a loss of experienced employees. Stack Overflow has tracked a persistent decline in Delphi activity and even found it was the second most disliked language when studying what users looking for jobs listed as languages they would not like to work with. Still waiting for this year's results, but the Stack Overflow user survey from two years ago showed 0.75% of users using Delphi, and that was with a push from EMBT for users to write in Delphi.

      If that's "alive and well... kicking and dancing", I'd hate to see what it would need to look like for you to consider it on life support. There are no objective, quantifiable measurements that show any signs of life for Delphi, while most show the opposite.

      In fact, Idera's Mr. Popov himself, in an interview with Blaise Pascal magazine, had this to say:

      "I think that the demand for a free Delphi Starter version may not be so high because there are fewer independent young developers. We have either people that have worked for a long time with Delphi and their companies can pay for the tools or students that will get access through our academic program."

      Interviewer: "That's not really true."

      Popov:"Possible, but that is our perception of the market today and that is why we have not focused so much on the Delphi Starter edition."

      So even Mr. Popov is tacitly admitting that there simply isn't an interest in Delphi among young developers today and hence they're not focusing their efforts there.

      Delete
  3. Well I was enjoying reading this (and agreeing with everything) until the bit about the language. The delphi language is decrepid at best.. it's in desperate need of modernisation.

    Perhaps it's time to get out of the IDE business and stick to compilers, and either write a plugin for another IDE or just let someone else do it. BTW, using seperate processes isn't the best idea imho, yes it works for VSCode, but then it has to, because it's a nodejs project and can't load up dll plugins etc.

    ReplyDelete
    Replies
    1. If by "modernisation" you mean "C#'ing", then I wholeheartedly disagree. I like the verbosity of PASCAL, and don't want it to be cryptic to read. A Pascal program can be read by a layman and (more or less) understood, whereas a language based on C and its derivatives is cryptic and needs a computer science master degree (more or less :-)) to be able to be read.

      Delete
    2. C# in no way cryptic to read. C and C++ maybe is.

      Delete
    3. Vincent you know better than to suggest an idea like a whole new IDE not even written by EmbarcIdera. Idera (and its Embarcadero division) either will or will not fund a commercial effort to modernize the IDE or the language. In my opinion, C# is beautiful but far too complicated for many users, and C++ is just an aberration of nature, which deserves to be destroyed. Opinionated? Heck yes. I love C#, and I enjoy working in it. But I like the Pascal language even more.

      I don't like the idea of people inserting things in there that don't need to be there. As pointy headed as the Google Go people are, I see the wisdom in their approach. No exceptions? No templates/generics? That's a bit harsh. But they drew a line. That's the part I agree with. Draw. A. Line.

      Delete
    4. Stefan I totally agree. C# is the most readable and nice language I know but there are crazy hairy elements of understanding it. Have you taught anyone how to write async/await style code? Have you debugged LINQ expressions? Have you had the full quota of fun with Entity Framework debugging? How about all the fun that comes with dependency injection? These things are not the language being hard to read, but they are C# as a tool, being harder to reason about than a tool which lacks these lovely features (that I love).

      Simplicity in features and in language are a benefit we give up the more we add to our IDEs and our languages. Delphi 1.0 was absolutely stunningly easy to learn, for example.

      Delete
    5. Entity Framework is a library and has nothing to do with the language - have you ever debugged FMX or any other over engineered framework? Same issue applies - not the languages fault.

      Async await - sure, every threading and parallel computing is complicated but I think you agree that its way more convenient and working than some manual hand written TThread stuff or half assed PPL.

      Debugging LINQ expressions - yes, that one can get quite complicated at first but if your brain does not grasp them, you can still write for loops like in the old days. And fwiw this is like saying chinese or russian is complicated just because I don't understand it - yet millions of people perfectly speak it.

      Delete
    6. Right. For what it does best, C# is the best thing in the world. For what it does best, Delphi is the best thing in the world. There is very little overlap. I don't like to write RESTful servers in Delphi. I don't like to write Win32 WPF or WinForms apps in C#.

      Delete
    7. Warren - you maintain that the language doesn't need improving and cite C++ as a warning. Yet hasn't Remobjects taken Delphi, added an arm's-length list of modern features, and still kept it as recognizable Pascal?

      No statically-typed language introduced since the dawn of the new century (which was 18 years ago - feeling old again) has omitted type inference (which was itself developed in the 1980s). Delphi's ancient single-pass compiler design (in Wirth's own words designed for a time when the memory of mainframes was measured in kilobytes) requires everything to be defined in sections. The only other language I know of still in use that has a similar requirement is... COBOL.

      We can talk about the antique features of the IDE causing potential new users to run away in fright (even Microsoft added git support before EMBT, and MS sells a commercial VCS), but the same is true of the language. There are simply language features that are considered "standard" today that are lacking in Delphi. The bloat and verbosity are also off-putting. When I showed a ruby developer what it was like to use REST with Delphi their response was "Yuck. I don't think I could work with a language like that." :-( Similarly when I explain to younger folks that Delphi has five (or is it more?) string types their eyebrows shoot up. And defining class methods outside the class itself also seems alien to them (Oxygene cleverly has made this optional now).

      I've often asked the optimistic Delphi users what technology or computer language has ever fallen out of favor and then made a resurgence. No one has ever had a good answer. I have a rebuttal to my own question - BASIC. However, in order to do so, BASIC had to acknowledge that the GOTO Wars were over and they lost. BASIC dropped the line numbers (its most iconic feature), added proper subroutines, etc. and enjoyed a resurgence from GFA BASIC and STOS in the Atari/Commodore days up to Visual BASIC.

      It's possible that a "Pascal++" could emerge to give us a better C++ that people are looking for (current candidates include D, Rust and Go), but it won't do so without being willing to acknowledge that, like GOTO, some of its features are evolutionary dead ends and need to be updated. The question is, would users be willing to accept this price? I know Stefan would, but I believe he's in the minority on this.

      As one example - you wrote that you consider anonymous methods a very ugly part of modern Delphi. I agree. The problem is - the language DIDN'T NEED THEM. You're a Python user. You know that Python doesn't have anonymous methods. Why? Like Delphi, it allows nested functions. If you can define a function right where you need it, there's little point to anonymous functions. Delphi's problem is because of that damn single pass design, we can't put those nested functions where we need them. Throw the single pass nonsense overboard, allow defining functions anywhere, add type inference, and BOOM! you can throw the verbose anonymous functions overboard too with no extra verbosity or loss of clarity. Then we can add lambdas, which is what we could really use instead.

      Just like the IDE could be improved if they were willing to invest the time and money and had the talent, so the language could be improved if they were willing to modernize it. As is, per Barry Kelly the Win32 compiler is over one million lines of undocumented (!!!) C code. No wonder there's no desire to make compiler changes! A rewrite of the desktop compiler could solve this... but then there's the different memory models to contend with. :-( It seems Delphi has Greek levels of (technical) debt, and this hinders any attempt to improve.

      Delete
  4. "Delphi is kind of beautiful and clean" -- I agree with that.
    But the IDE certainly needs to look fully Windows 10 when running on Windows 10.
    For example, the Form Designer -- It's a small thing, but...

    ReplyDelete
  5. شكرا وارين .. أحببت المقال.

    ReplyDelete
    Replies
    1. You're welcome. I didn't know how to read Arabic but Google Translate helps.

      Delete
  6. FWIW, did you know that when that dreaded dialog comes up telling you that it can't find a component, it offers the options to remove, ignore or cancel. Just "press" the Cancel button and NOTHING WILL BE DELETED. You won't be able to bring up the designer for that form, but everything will stay in place. Only the two other options would remove the component. So install the components, close the form file and re-open it, and everything is fine again.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Hi Rudy. Thanks for trying. If there was only ever ONE of those components and if you were expecting that it might come up, you might know that clicking Cancel is "don't break my forms". No application should have a dialog box at all where any of the choices, answered yes, no, or okay, or cancel, causes it to break my app. It was a horrible wrong, bad decision, and was a mistake.

      Delete
  7. My favorite points are 1.1, 4, 5. Does the whole IDE need to be reworked to achieve? Don't know. I like the latest iteration of the IDE. Pretty solid too. My wish is for more intelligence, i.e. ReSharper features.

    ReplyDelete
    Replies
    1. Guess what powers the ReSharper features. A language server engine. I have thought about that a lot. Adding anything to the existing code completion engine would stink because the whole code completion engine fails to work at all on many large codebases. The refactoring engine has its own parser, and fails to work at all on many codebases. See the common theme? A chaotic array of antique sub-components that don't work together.

      Delete
    2. Guess why Microsoft put years of work into the Roslyn project - because they wanted to get rid of 2 separate compilers for C# (one for actually generating programs and the other for IDE tooling).
      And no wonder Typescript follows the same concept. The day they add a new feature every tool using the language server knows about it.

      Delete
    3. Exactly right Stefan. Glad to see someone else who was following what Microsoft accomplished in Roslyn.

      Delete
  8. IntelliJ is so far ahead of the game is not even funny.

    ReplyDelete
  9. I agree with all points and if you look at the price for the IDE and what you get ... In my opinion,this is out of proportion.

    ReplyDelete
  10. Sadly, Idera/Emba sends out marketing surveys that don't really give them a way to see the importance of something like this. I fill them out, and tell them yes, VCL matters a lot to me, etc etc -- but if I were able to rank the importance of new development on Delphi, I would tell them that the IDE itself is the most important thing. I have a moderately large project that uses interfaced objects a lot, and that seems to completely break code completion and error insight easily. When I start a new small project and code completion is working as it should, it seems great, and I keep wishing it would always work -- but as soon as the project grows and a bunch of interfaces start getting used everywhere, code completion becomes a crap shoot.

    And Warren is absolutely right about this thing being fundamentally broken when it comes to damaging DFMs. Rudy acts as if the dialog so politely asking to cancel is enough -- but anyone who has ever had to migrate a large, non-trivial project group that contains hundreds of components knows it is broken and wrong.

    ReplyDelete
  11. By the way, here is a QA report I submitted about code completion: https://quality.embarcadero.com/browse/RSP-19116 notice that I just pointed them to VCL.Controls as an example that breaks it. Does it seem odd to anyone else that they seem oblivious to the fact that code completion doesn't work on their own code base?

    ReplyDelete
  12. Warren - great article! It often feels to me that there are several people in the Delphi community who could take over and run the Delphi development a lot smarter/better. You've always been one of those people to me. I'd like to see Stefan turned loose on the language itself though. ;-)

    That postscript you added is very insightful and I was planning on offering thoughts along the same line. I'd go a bit farther though. I've often heard the users fearful of any language changes express the sentiment that they want to keep their codebase the same but at the same time want to upgrade to take advantage of IDE improvements. So long as the IDE is being (hypothetically) remade, why not go even farther in decoupling?

    If the IDE itself were not tied to one particular version of the compiler/linker/parser/form designer then one IDE instance could handle projects from many different versions of Delphi! You bring up being able to have components and other options on a per-project basis. In JetBrains' Python IDE the Python interpreter itself (along with a virtual environment) is a per-project option. Hence the same IDE instance can work with projects developed with many different versions of Python while properly handling code completion, refactoring, etc. This would be wonderful for Delphi and make a heck of a lot more sense than having multiple versions of Delphi cluttering up one's disk and needing to hop between them (and each being different in terms of bugs in the IDE).

    I'd hoped that GetIt would have changed Delphi deployment, but it's a poor package manager... as far as I know no web interface, CLI, ability to work with third-party repositories, etc. After years it still only has 200-some packages and won't allow prominent Delphi languages such as Mormot for silly reasons of "competition" making it useless to easily replicate Delphi projects across instances. Delphi developers are using virtual machines and all sorts of other uncommon methods to deal with the difficulty of deploying a new developer instance. The decoupling changes combined with a real package manager would practically eliminate this burden.

    JetBrains' installation is also much simpler and safer than Delphi's. They now have a cross-platform app, Toolbox, that handles it. You can sign in and then install any of their IDEs you own, any version you own, or a demo version of one you don't in a single click! Upgrades also show up here and a single click can upgrade all of your installed IDEs. Better still, it installs the new version then attempts to import the settings from the old version. Only if successful will it then delete the old version and rename the new instance. Your settings stay safe. With Delphi one uninstalls first and then hopes the new install is successful - if not, you no longer have a functioning Delphi.

    While Delphi has an option to export IDE settings, JetBrains again takes this one better. They actually offer cloud storage space on their server for IDE settings! When you install on a different machine via the Toolbox it can pull down those saved settings too, obviating the need to configure the new instance manually or carry around a possibly out-of-date exported settings file.

    Delphi has a long, long way to go in terms of being able to replicate setups and making installs, updates and project replication easier. It was eye-opening recently to move a Python project from one system to another. A single command to the package manager on machine A was enough to produce a text file I could feed to the package manager on machine B to replicate all of the libraries and dependencies. As you noted, I just needed this one file and my code itself. JetBrains' PyCharm was also able to create a new project, read the new "virtual environment" and set everything up correctly - even across different operating systems!

    ReplyDelete
    Replies
    1. I like your ideas, as long as they reduce learning burden and cognitive complexity for developers. Jetbrains has done a lot to help folks in that area. They need to do EVEN more now that Java 9 and modularity has hit the JDK.

      Delete
  13. alcade says: " Warren - you maintain that the language doesn't need improving and cite C++ as a warning. Yet hasn't Remobjects taken Delphi, added an arm's-length list of modern features, and still kept it as recognizable Pascal?"

    Please write a blog post about how beautiful Oxygene is, because I haven't spent the time to really know either way. All I do know is that the older and grumpier I get, the more I'm turning into Rob Pike. YAGNI is the best language design.





    ReplyDelete