Monday, December 17, 2018

GetIt is broken, please fix it Embarcadero

Various people, including myself are having problems with GetIt returning license errors instead of installing the thing it should install. Even on free items like the Jedi JVCL library, you can get this error:

It's not a blocker when you can install Jedi yourself from source (clone the git repo, download a zip from sourceforge, but where it really affects the product is when you build an Android firemonkey app the first time. You get a prompt to install the tools with getit, and getit having failed, your only choice is to uninstall delphi and reinstall from the ISO, instead of from the web installer.

I hope Embarcadero fixes this soon. If anyone knows a workaround to get Getit to function, I will edit this post, and I will update it once a fix is released.  Above is with Delphi 10 seattle, which many projects I work on are using still

Sunday, July 22, 2018

Delphi starter is back as "Delphi Community". Good news for all Delphi users.

I am one of those codgers old enough to remember Delphi 6 personal, and the Turbo Delphi era (the next free edition name for delphi), and Delphi Starter, and each time the pattern is the same:

1. Announce it.

2. Hope it works.

3. Wander away and lose interest.

Let's hope that DOES NOT happen this time. Delphi needs this "free" starter edition.  I wholeheartedly agree with the wiser and saner heads who will from time to time argue that this needs to come back if it's currently in a hiatus. Not dead, you understand, just resting, like that parrot in Monty Python.

With or without a free edition DELPHI IS NOT DEAD, but with a free edition it's healthier. We need a tools ecosystem, that's an essential part of modern software development. A healthy ecosystem needs us to be able to build free stuff for Delphi.  With a free edition of Delphi, I hope it will always be possible to run continuous integration servers, for example, for open source projects written in and for Delphi development, and not need to worry "how do I build and run its unit tests and deliver the installers for our product/package". 

We need encryption libraries. We need REST server and rest client frameworks and add-ons thereto.  We need something approaching the breadth and depth of the ecosystems of Java and .Net, if Delphi is to continue to exist as a sensible way to work in a web-everywhere connected world.

So delphi Starter, ahem, Community, is back. I for one am glad.

Friday, February 23, 2018

Windows 10 Is Terrible

You would think that Microsoft, and Apple, having some of the highest paid software engineers in the world, would generally deliver higher quality software, that makes its users happy.

You will notice I didn't say Best Software Engineers.  I think that such a term is meaningless. We are all human beings.  The things we produce have bugs. And worse than that, because bugs were accidents, we hope, there are misfeatures. There are misbegotten elements in Windows, and Mac OS, that you cannot alter. They are part of choosing to use these products.   There are serious problems in Linux, but there are literally no misbegotten elements in Linux that cannot be removed. You can literally do anything to Linux.

But since most of the professional world runs on Windows,  and most of the Creative World (including most of the music workstations, and pro digital photography world) lives on Mac OS, both mac and windows users have a common problem.  You probably don't know, you even probably don't want to know, what your operating software does,  and whether you do, or you don't, you can't fundamentally control it.

I will write a separate post on my Linux focused blog on why Linux is terrible, but since most of my Delphi blog readers are, like me, stuck using Windows, either on your PC, or in a VM, this rant is mostly about why Windows 10 is terrible, and why staying on Windows 7 forever isn't an option either.

1.  You don't actually control your computer anymore.    

1.1  You have no way to stop system reboots.  Hello, Microsoft Here. We've decided to reboot your machine.  You can't say no.  This is acceptable?    Microsoft will of course counter that they give you some control now, you can say, well I'm working from 9 to 5, don't reboot then.  And you can say I use my computer actively these days. You can't say, no, let me decide when to reboot.   It kills me that even Microsoft employees get Windows 10 reboots while they are giving presentations on how great Windows 10 is.

1.2  You can not disable Windows Defender, unless you want to install something worse.   You can temporarily disable the Windows Defender Realtime feature, but it is automatically re-enabled after an unspecified amount  of time. From what I can see that time is about 1-2 hours.  The service that you could have disabled in Windows 7 to prevent Windows Defender is no longer possible to disable unless you take extraordinary steps to grab permissions of registry keys, and do some low level hacking that could easily damage your Windows install.

1.3  You can not keep permissions on NTFS files anymore.  If you, like me, think that the C: drive root can and should contain folders that are writeable by a non-admin user, Microsoft  disagress. They reset all folders that start with the root of the boot drive as part of the regular security sweeps of your system, and you can not disable this. This breaks several applications that I maintain and develop.    It seems these sweeps are associated with windows updates, so when Windows 10 updates to a new kernel, it does these NTFS permission updates.

1.4 You can not be sure you'll have 100% of your CPU and disk bandwidth available, at any time. Who is using all my computer's resources right now? Oh yeah, Microsoft. There are many privileged tasks that  you can not fully control or stop, including superfetch, the trustedinstaller, and there are many that do not tell you they feel like using your whole computer that at least can be paused.  The windows search feature, which is useful, is not configurable so that it doesn't use your valuable computer resources when you're trying to, say, write code and compile your applications in Delphi and then debug them.

2.  Windows 10 is gathering information about your computer, and you don't know what it is.

Some information has been revealed about what kind of data Microsoft collects, but you still can't opt out of this, and you don't know what data it sent from your computer. In the latest Creators Update you have some control over the level of data gathering, but Off is not an option.  This is frankly, unacceptable.  On my main Windows 10 box, the Privacy page in the Settings app, has some new options, but "opt out from all telemetry and data transmitted to Microsoft" is not one of them.    I have to say, it is nice that there is a web page where you can view the data they have that is associated with any email address you may choose to log into that site with.  But even they (and you) may not actually be aware of, and may not, via this web site, be able to say "we're not monitoring you".   They can only say, "here's data we have for that microsoft account, associated with that email".   

They do not claim that the data you clear is everything.  In fact, the EULA makes it clear that it's not.  You can clear out your browsing history but not all your telemetry data.  Why do I even care? Because it's a principle.  Surveillance that is used to send your CPU load history, that can be used to see performance regressions, arguably helps Windows users. But it should be my choice to opt into even the most benign data collection programs.   That your browsing history (for Microsoft Edge only) is clearable, is good.  All they give you there, is good, assuming you trust them. It's just not enough. It's sort of a smokescreen.  I'm not saying that they just pretend to delete data, but I am suggesting that the real picture might be a lot more complex than they want to admit publically. They don't actually know who had access to personally identifiable data, that may do something with it, that is an abuse.   They can't because that's a human problem.   The best procedure for privacy is to not collect data in the first place. Microsoft has become addicted to your data.  It's their lifeblood, they admit that publically.  They say it's the "lifeblood of Windows". No, it's the lifeblood of the new Microsoft.  Your life.

So why not stay on Windows 7? The ecosystem will continue to rot, and eventually it will become difficult to run Windows 7 on modern hardware. Microsoft has even started actively looking for ways to make Windows 7 not runnable on new hardware.    Windows 7 is almost as bad as Windows 10, but at least, when you disable the Realtime mode in Windows Defender it stays disabled.  Windows 7 is not ultimately a good solution.    If you have Windows 10 Pro, you might find that the Group Policy editor provides a "close enough for rock and roll" method for disabling Windows Defender, via group policy. If you have Windows 10 home, that method is not available.  Windows 7 has many of the same flaws (superfetch, boot time problems, update problems, windows defender problems) as Windows 10, but overall the symptoms are not as bad.  The most egregious problems in Windows 10 are somewhat less of a problem in Windows 7.  

What is the solution? I don't have one.  I'm just mad.

Thursday, February 8, 2018

The Concise Delphi Coding Standard

The Concise Delphi Coding Standard (2018)

1. Use the Delphi Source Code Formatter on New Units before you Check them In.  When you touch a function keep its formatting style the same, or reformat the whole function with the Source Code Formatter. Only that one function you touched. Use your team's settings which are shared within your team.

2. Do Not Randomly Reformat Entire Files because of Coding Standards.  If you need this point explained to you in detail, you need a lot of re-education about version control.

3. Avoid WITH 

4. Avoid obfuscation, including unnecessary accidental complexity, avoid unreadable code, very long functions over 80 lines in length are completely unreadable, and are a form of obfuscation, as are placing things far apart that shouldn't be far apart. Avoid that.

5.  Bend, don't break. Do not start holy wars over spacing, indentation, and variable naming, or any other thing in any coding standard ever.  Find a peaceful way to defer like everybody defers to the Graybeard, or whatever.  Just bend, and go with the team. Do not turn coding into an emotional bloodsport.


There's actually only one thing in the above list that's explicitly delphi specific.  But the VCL source code formatter's default settings are pretty reasonable.  I prefer to add line breaks to uses clauses as that enhances readability, reformatting the following into one unit per line, which with the addition of Delphi namespaces makes Delphi uses clauses a lot more like reading the import list of a java or C# module:

  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls, Mask, ExtCtrls, AdvGlassButton;

Wednesday, January 31, 2018

How to Bring Delphi Back from Life Support

I was recently commenting on Facebook and asked what could the community do to help Delphi get more popular again.  My answer? Uh Nothing. 

A real change would start with the product actually getting a serious refresh, something probably that nobody can afford to do, and which EmbarcIdera is extremely unlikely to fund.

Cue up a nostalgic 80s song, maybe Glory Days, by Bruce Springsteen.  Grab a beverage. I'm about to rant.

The Glory Days of Delphi were from the Delphi 5 era until Delphi 2005, where the product foundered severely. 2005 was so bad, and 2006 was only slightly better, that few people could use it. To this day I encounter companies still using Delphi 5 or 6 or 7 because of this ancient developer and QA failure made in the tail end of the Borland era.   Let's face it. Shipping Delphi 2005 was the worst thing that the management ever did to the product's reputation. Even worse than the well known shenanigans at their Sales level where they were pushing their (now dead) Java IDEs on their existing Delphi accounts.

The blowback from the instability of 2005 was actually less than the blowback for the UI changes. Developers (it me!) are a surly bunch.  You moved my buttons. Damn you!

Now the UI changes were well intentioned, and well done, and were intended to bring Delphi out of the Win3.1 era UI of Delphi 5 through 7, and into a docking style IDE, comparable with Visual Studio 2005. I like the new UI in 2005+ delphi editions better than the old UI, but the point is that the old IDE was more stable.

There has not YET been another full refresh of the IDE.   Delphi has continued to be usable, and a good productive tool, and while it's IDE architecture continues to look worse by comparison with its one time equals, it is now so far behind it's not even funny.  Remember there are people using FoxPro and Visual Basic 6 IDEs and loading them up in Windows 10.  We're more like them, than different, at the moment.  The registry key where delphi gets installed gets a regular major version bump, we're at what 18.0 now?  But the IDE architecture is frozen in time.

The bds.exe code of 2018 is mostly the same in operation, and contains most of the same assemblies and parts (binary dlls) as the bds code of 2005, with the addition of creaky refactoring code built in J# (yes JSharp, that no longer supported .net 2.0 language), error insight that doesn't work (due to having its own parser which doesn't parse all the language features, or even the compiler search path behaviour of the real compiler). The editor core in C++ is buggy to this day and crashes, and loses people's work. 

A lot of talented people worked really hard and fixed a lot of bugs, and Delphi XE8 and up, including the current Delphi 10.x series are very stable IDEs. Don't get me wrong. I appreciate that work.

But what we have here is a polished antique.  The editor core is basically the code from Borland C++ for Windows' editor, from about 2001.  The code is a legacy codebase, and it needs a total IDE overhaul.

The delphi DCC32 and DCC64 compilers are relatively good compilers, excepting that there is no significant OPTIMIZATION available for DCC64 still.   If you are familiar with the level of optimization work done in modern C++, and you compare it with Delphi in 2018, you're gonna be sad.  I think that the Nextgen picture may be significantly more rosy with LLVM's optimizers at work, but that's never going to be an option when Nextgen was designed from the ground up to contain many breaking changes to your code, including string handling, and other fundamentals.  That's not gonna be pretty.

But the Compilers, classic and nextgen, produce really fast code, even though they are not at parity with say, the Visual C++ 2017 optimizing compilers.    I'm actually fine with that. I actually still kind of  hate C++.  Pascal is a beautiful language.

The bigger problem is the IDE is not very good. 

Spend 16 full hours getting work done (pair programming) with a colleague who knows what they are doing in Visual Studio (perhaps with resharper or not), or IntelliJ IDEA and you will realize that what we mean when we say "IDE" in delphi is at a point similar to what Most IDEs Were Capable of Circa 2005.

Alright Smart-Ass What would You Do Differently?

Fair question.  If you gave me a 6 million dollar development budget and two years I would rearchitect the IDE, but not the compiler, or the language specification:

1. Subprocesses are key. Yes, yes it will use more memory. It will crash less. Are you sold yet?

1.1. Make the editor core a subprocess (RADStudioEditor.exe). Less crashing.  Large file search/replace should even be able to happen in background in parallel.

1.2. Make the compiler and code completion engine a subprocess (RADStudioLanguageServicesEngine.exe).  Kibitzing compilers, code completion and error insight compilers would share a single tokenizer/lexer and grammar.

1.3. Make the IDE main window (shell) a lightweight shell written in Delphi (RadStudio.exe).
4. Make the form designer also a subprocess (RADStudioFormDesign.exe). This subprocess would be capable of loading a set of BPLs needed to open the form.

2. First do no harm. Don't Tread On my DFMs. I would fix the ancient frustration caused by failure to locate and install all BPLs that a form requires before you open a form.  It would be possible to open a form without the damn thing DELETING YOUR COMPONENTS.   Similarly, all the places where it asks you sixty million times in a row are you sure you want to delete your components would default to no, don't bug the user, and don't do destructive stuff, instead fall back gracefully and do nothing.  Those fun times when you have to open task manager and kill delphi to get it to stop asking you 9 million questions. Good times.

3. Stability is key. When a BPL fails to load or segfaults, instead of the IDE going down, you would lose one DFM editor tab.  

4. Modern editor features.  Delphi's next IDE needs navigation and quick look features on par with IntellIJ IDEA and Visual Studio 2017.

5. Unified compiler and code-completion (kibitzing) engines, with a single root grammar and Lexer.   The compiler and the code insight engine need to work similarly to the modern architectures used in Visual Studio and Visual Studio Code.

And a few things I would NOT do:

1. We Don't Need No Stinking New Language Keywords because We Are Not Jealous of C++.  The language in Delphi is actually it's most important asset. Adding crap to the language is not gonna make it better. Delphi is kind of beautiful and clean, like Google Go.  I actually think the syntax of anonymous methods is the ugliest part of the current language spec. I'd like to stop now before we end up with crap like they have to deal with in the latest C++ language spec.  Delphi's module system, and runtime plus compile time typing system are better, in my opinion, than C++'s mostly compile-time-only type systems, and current lack (in 2017) of modules. I don't want to copy C# either.

2. Break stuff. I would not break anyone's code, DFMs, .pas files, etc.  Absolute compatibility is the only way to guarantee you can upgrade.  The decisions already made to break stuff in nextgen, while I understand the difficulty that lead to the reluctant change, I still thing are going to be the undoing of the entire thing.

Would 6 Million American Dollars be enough budget to do all the above? Would it take 2 years? That I don't know.  I just know it would take a lot of work.     And given that the business case for the above might be that the payoff would be around $4 million dollars, it remains probable and possible that doing all the above is a business mistake.  

 What I do know is that the above is the list of things being NOT fixed, are among the things that would make most young developers who know any other language, and even most old farts like me, want to Stay The Hell Away from the product.

When Delphi works, great, it's a joy to use.  A proactive solution to the long term trouble with Delphi stability requires a modern multi process architecture, and a significant R&D commitment.  It would also be nice if you still had some key developers who knew where all the bones are buried.   If you didn't have all that, I'm just saying, that maybe six million dollars is still not enough to do the work above.

The market realities today are stark. You can get a trial of IntellIJ IDEA for free and run it for months, and you only pay a few hundred dollars for it if you buy it.  Visual Studio Community is freely downloadable and is the default choice for Windows-Centric developers today, which is funny because WinForms and WPF still suck.   

Delphi is a niche product today, an important and interesting one, but it's probably never going to be even 10% as popular as the big boys, ever again.  I still think it's a great thing to build Win32 apps in, and I do wish it wouldn't eat my DFMs and my code on me.

Rant over. [Mic drop]

Postscript:   A very senior Delphi Developer where I work pointed out that another core Delphi IDE design mistake is that the Library Path, and Component Installation Registry keys, and other similar items, exist at all, at a global level, which is not committable to source control, but which does affect your ability to open, share, and build codebases in Delphi on multiple developer machines, and that these settings should be  per-project and be as files on disk, not in the registry. I agree with him heartily, and feel he was right when he said that is perhaps the single worst problem with the current IDE design.  If I were to replace the existing crap, I would introduce a concept like Workspaces in most java IDEs, such as  IntelliJ IDEA, to the Delphi world.  A copy of the files and their workspace should be all you need to transport and build Delphi code anywhere.  And it should be possible to load different versions of different components into different projects without the egregious workaround of bds.exe -Rsomething

Tuesday, September 12, 2017

Agile is Dead. Long Live Agility and Pragmatism.

The luminaries who signed the original Manifesto for Agile Software Development in the 1990s have come back more recently, including Dave Thomas who said that "Agile is dead" in 2015, as part of his reflection on what went well, and what didn't go well, in the Agile Software movement they started.

As often happens, practices which had an intent, a purpose, a meaning, and an effectiveness, can lose that purpose, meaning, and effectiveness. As Dave Thomas laments, the Values get lost behind the Implementation, and Agile Software Development, shortened ludicrously to just "Agile", has become that which it mocked, and sought to replace.

That being said,  I actually appreciate the core values that underpin the Agile manifesto, even while I  advocate a skeptical approach to all methodologies.  Don't drink the kool aid.  Question.  Gather data.  Use Data and your ability to Reason. You can be a skeptic of ALL methodologies and also be a bit of a proficient practitioner of one of them, Agile or another.

In my current team, we're actually doing a really light set of "Fairly Agile" practices.  I've been on this team nine months now, and I really like the way we work.  The things we do are things that give us lots of positive payback.

Here they are:

1. Code Reviews

Shipping code is reviewed (hopefully actually read) by more than one developer. We use a distributed version control system (git) and pull requests to do our code reviews. We use bitbucket.  In my previous job we used git with gitlab, which also worked great.

2. Light Daily Standups

Our daily standups are in Slack (text chat) 3 days a week, and are video two days a week. Our team is mostly remote and disbursed.     The video standups are much longer than the usual standup time.  But since the team is distributed, and there are relatively few other meetings, we are able to spend a lot of each work day building things.

3.Lightweight "Paperless" Audit Trails : Slack All The Things. Don't Email.

We don't ship changes to code that aren't tracked in an issue tracker.  We don't make discussions verbally, or in emails.  We default to public place accessible by all employees (slack chat) for all architecture and UI changes.  Emails are private and are thus problematic for the team, or else turn into an unsearchable tar pit, and may as well have never been sent, as you can never find anything.

 4. Pair up when you need it.

We have a practice of teaming up with another developer whenever that would help get us unstuck, or do something that one person alone couldn't do.  We don't do that frequently, but we don't do it NEVER. We do it when it works.

I think these practices are great.   I don't think we're doing everything perfectly.  I think a certain amount of discontent is actually a core value for pragmatic/agile/high-quality software developer work.  For example,  I wish our codebase let us write more tests. That's something we probably can get to when our codebases are decoupled enough to permit unit testing.

What is your core "Agility" and/or "Pragmatic" practices list? What do you think makes  your software team work effectively?

Cool Advanced Troubleshooting Technique - WinDbg

I was having some problems with my Thinkpad laptop hanging. It didn't blue-screen. It just stopped updating the screen, and did not reboot. It seemed either that the entire video layer had hung, or that the kernel itself had hung (such as waiting on a spinlock forever), or that a critical subsystem of my computer had failed (since windows uses virtual memory heavily, such a problem can also be commonly caused by a storage layer failure, such as your primary boot hard disk).

What's relevant to this delphi blog,. is that it was Delphi's own debugger that most commonly triggered the hang.  Exiting the debugger, terminating a debug session, was the most common way to hang the system.

I've watched a few of the videos on MSDN Channel 9 from the guys called Defrag Tools. This is some serious low level technical know-how that these folks are dishing out. If you're into it, go check them out.   

Anyways after a bit of troubleshooting, I decided to try switching out various driver versions to see if I can make the hang go away. After trying the OEM Intel HD 520 graphics drivers from Intel's website, I was able to work inside Delphi all day today with NO crashes.

This is not my first "Delphi + Video Driver" hell experience.   About two years ago, I had a crazy set of bugs in Delphi that only reproduced with certain ATI/AMD video cards, and only in parts of my Delphi application using Microsoft Direct2D/DirectX.   Direct2D surfaces would just stop rendering, and whatever parts of my delphi application were using Direct2D apis just stopped working.  A couple years before that, I saw similar problems with GDI+, also on ATI/AMD video cards.   And going way back about 20 years ago, in the WIndows 98 era, I remember there were bugs in ATI video cards of the era that broke the ability of the Win32 GDI layer to render transparent bitmaps.

But this one is the first time I've had a Delphi program, or Delphi itself, plus a bad video driver, actually hang my entire system, requiring a cold reboot to recover.

More details on my question on here.

Some suggested ways to get started:

1. install the Windows 10 SDK to get WinDBG installed on your computer.
2. on a system which generates BSODs, enable memory dumps on your computer in your System -> Advanced Startup Options.  In systems which hang without a BSOD, learn how to enable the Scroll Lock key way to force a hang.
3. watch a few Defrag Tools episodes to get the flavor of how to use WinDBG.

Note that full memory dumps are huge. If you have 64 gigs of RAM a full dump will be more than 64 gigabytes in size.  On my 1 tb disk with about when my total free space is less than 16 gigs, a Helpful windows feature will automatically delete my dumps so that windows doesn't fail to boot.   Try to have at least twice your total memory free on your main hard drive before you try enabling full dumps.  If you have 64 gigs of ram, make sure your primary windows boot disk has at least 128 gigs of free space. For just a bit of learning with Windbg, just enable some non-full-memory dumps so you can get the flavor of the WinDBG tool, and try entering some commands.

There are two "sides" to WinDBG, the kernel side and the user space side.  In the kernel side, you can view call stacks, thread states, change which CPU you are looking at, list NT native kernel level processes, and walk back in time to various saved states. In the user space side, you can see the  Win32 level APIs, processes, and walk back to a time prior to the last saved exception to examine the state of your system before the last first chance exception.  Besides being able to see C/C++ symbols from non-managed DLL/exe images, you can also use some extensions to be able to see things from inside the CLR, so you can troubleshoot problems with .net code.

Unfortunately, the debug information format used by WinDBG is not the same debug information format that is produced by Delphi so this tool is not very helpful for debugging Delphi application crashes, unless you don't mind working with uninstrumented disassembled code with no debug symbols.